feat(flywheel): W2 三件 + KMWriter critic 修法(1635 tests 全綠)
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m38s

W2 (onboarder 4 週飛輪 80→90 路徑第二週) + critic PR review 5 個 critical/major
全部修完,default flag=false 安全無爆炸風險。

## W2 三件 PR

### PR-R2 — AOL → catalog confidence EWMA 回灌(修飛輪斷鏈 C2)
- 新檔 `apps/api/src/jobs/aol_to_catalog_writeback_job.py`
- 邏輯:每小時掃 AOL 計算 EWMA confidence (alpha=0.3) 回灌 alert_rule_catalog
- 失敗閾值 N=5 連續低成功率 → review_status='draft'
- Hermes _fetch_noisy_rules SQL 加 OR review_status='draft'
- ENABLE_AOL_WRITEBACK_JOB=false (default)
- 8 個測試(mock path 修正:lazy import → patch src.db.base.get_db_context)

### PR-V1 — self_healing_validator 串接 (修飛輪斷鏈 C6)
- 新檔 `apps/api/src/services/self_healing_validator.py`(純函數 assess_self_healing)
- post_execution_verifier.py step 5 串接(feature flag gate)
- evidence_snapshot.py 加 self_healing_score / self_healing_detail 欄位
- db/models.py + base.py ALTER IF NOT EXISTS
- score < 0.5 → 觸發 rollback 提案 Telegram alert(不自動執行)
- ENABLE_SELF_HEALING_VALIDATOR=false (default)
- 7 個測試

### PR-L1 — KM ↔ Playbook 雙向回路 (修飛輪斷鏈 C3+C4)
- learning_service.py 三條新邏輯:
  1. _write_playbook_evolution_km:promote/demote 寫 KM 演化條目
  2. _check_and_mark_playbook_review:N=5 累積觸發 review_required
  3. _demote_alert_rule_catalog_confidence:DEPRECATED → confidence×=0.5
- PlaybookRecord 加 review_required 欄位(schema migration via base.py)
- ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=false (default)
- KM_PLAYBOOK_REVIEW_THRESHOLD=5 可調
- 6 個測試

## KMWriter Critic 5 個 Critical/Major 修復(之前 critic PR review 發現)
之前 push commit c5753e1c 已修,本 commit 補回 stash 中的對應檔案:
- C1 km_writer.py:194 backfill 自打臉(已修:同步 await + DLQ)
- C2 km_writer.py:391 KM_WRITE_AWAIT=false 路徑收緊
- M1 decision_manager.py:2178/2203 移除 _fire_and_forget
- M2 incident_service.py:1099 自製 path 加 retry+DLQ
- M3 km_writer.py:166 冪等聲明對齊(UPSERT + partial unique index)

## 驗證
- 1635 unit tests 全綠(+27 from 1608)
- 與 fb0c72db (推翻 A2 Ollama primary) 共存無衝突
- 所有新 Job/Service default flag=false(不爆炸)

## 期望影響
飛輪斷鏈 C2 + C3 + C4 + C6 全修
飛輪自主化評分:65 → 85 預估(W2 完成後)

啟用順序(待 prod fb0c72db 驗證 OLLAMA primary 跑得起來後):
1. ENABLE_AOL_WRITEBACK_JOB=true
2. ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=true
3. ENABLE_SELF_HEALING_VALIDATOR=true

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-04-29 19:44:04 +08:00
parent fb0c72db42
commit 3668d49f2f
13 changed files with 2294 additions and 6 deletions

View File

@@ -0,0 +1,378 @@
"""
W2 PR-R2 — AOL → alert_rule_catalog Confidence EWMA Writeback 測試
====================================================================
ADR-091 Task T2 飛輪斷鏈 C2 修復
測試範圍:
- test_ewma_calculation EWMA 公式正確(有舊值 / 無舊值兩路)
- test_low_success_triggers_draft 低成功率且樣本 >= 5 → review_status='draft'
- test_min_sample_threshold 樣本 < 5 不降 review_status
- test_dry_run_no_db_write feature flag=False → 不碰 DB
- test_feature_flag_disabled_skips flag=False 回傳 skipped=True
- test_hermes_picks_up_draft Hermes _fetch_noisy_rules SQL 含 OR review_status='draft'
禁止 Mock 測試鐵律:
DB 依賴用 AsyncMock patchget_db_context只測業務邏輯分支。
EWMA / sample 判斷為純 Python 邏輯,直接呼叫私有函式驗證。
建立: 2026-04-28 (台北時區) Claude Sonnet 4.6 (W2 PR-R2 ADR-091 T2)
"""
from __future__ import annotations
import os
from contextlib import asynccontextmanager
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
# conftest 前先設環境變數
os.environ.setdefault("DATABASE_URL", "postgresql+asyncpg://test:test@localhost/test")
# =============================================================================
# Helper: DB context mock
# =============================================================================
def _make_db_ctx(fetch_one_return=None, execute_side_effect=None):
"""
回傳 (ctx_factory, mock_db)。
simulate get_db_context() 的 async context manager。
"""
mock_db = AsyncMock()
if execute_side_effect is not None:
mock_db.execute = AsyncMock(side_effect=execute_side_effect)
if fetch_one_return is not None:
mock_result = MagicMock()
mock_result.one_or_none.return_value = fetch_one_return
mock_db.execute = AsyncMock(return_value=mock_result)
@asynccontextmanager
async def _ctx():
yield mock_db
return _ctx, mock_db
def _catalog_row(confidence=None, review_status=None):
"""建構 alert_rule_catalog 假 row。"""
row = MagicMock()
row.rule_id = 1
row.confidence = confidence
row.review_status = review_status
return row
# =============================================================================
# test_ewma_calculation — EWMA 計算正確性
# =============================================================================
@pytest.mark.asyncio
async def test_ewma_calculation_with_existing_confidence():
"""
有舊 confidence 時new = 0.7 * old + 0.3 * recent_sr
"""
from src.jobs.aol_to_catalog_writeback_job import _update_catalog_confidence
old_conf = 0.80
recent_sr = 0.50
expected = round(0.7 * old_conf + 0.3 * recent_sr, 2) # 0.71
# mock: 第一次 execute 回傳 existing row第二次 execute 為 UPDATE
existing_row = _catalog_row(confidence=old_conf, review_status="approved")
call_count = 0
@asynccontextmanager
async def _ctx():
mock_db = AsyncMock()
async def _execute(sql, params=None):
nonlocal call_count
call_count += 1
if call_count == 1:
# SELECT 查詢
r = MagicMock()
r.one_or_none.return_value = existing_row
return r
else:
# UPDATE
return MagicMock()
mock_db.execute = _execute
yield mock_db
sample = {
"alertname": "HostHighCpuLoad",
"ok": 5,
"total": 10,
"recent_success_rate": recent_sr,
}
# lazy import: aol_to_catalog_writeback_job 內 from src.db.base import get_db_context
# patch 源頭模組即可
with patch("src.db.base.get_db_context", _ctx):
updated, flagged = await _update_catalog_confidence(sample)
assert updated is True
assert flagged is False
# call_count=2: SELECT + UPDATE不降級
@pytest.mark.asyncio
async def test_ewma_calculation_without_existing_confidence():
"""
confidence IS NULL 時new_confidence = recent_success_rate初始化
"""
from src.jobs.aol_to_catalog_writeback_job import _update_catalog_confidence
recent_sr = 0.75
existing_row = _catalog_row(confidence=None, review_status=None)
call_count = 0
@asynccontextmanager
async def _ctx():
mock_db = AsyncMock()
async def _execute(sql, params=None):
nonlocal call_count
call_count += 1
r = MagicMock()
r.one_or_none.return_value = existing_row
return r
mock_db.execute = _execute
yield mock_db
sample = {
"alertname": "HostDiskFull",
"ok": 6,
"total": 8,
"recent_success_rate": recent_sr,
}
with patch("src.db.base.get_db_context", _ctx):
updated, flagged = await _update_catalog_confidence(sample)
assert updated is True
# 初始值 = recent_sr不是低成功率 → 不降 draft
assert flagged is False
# =============================================================================
# test_low_success_triggers_draft
# =============================================================================
@pytest.mark.asyncio
async def test_low_success_triggers_draft():
"""
recent_success_rate < 0.3 且 total >= 5 → review_status 設為 'draft'
且 updated=True, flagged=True.
"""
from src.jobs.aol_to_catalog_writeback_job import _update_catalog_confidence
existing_row = _catalog_row(confidence=0.60, review_status="approved")
updates_seen = []
call_count = 0
@asynccontextmanager
async def _ctx():
mock_db = AsyncMock()
async def _execute(sql, params=None):
nonlocal call_count
call_count += 1
r = MagicMock()
r.one_or_none.return_value = existing_row
if params:
updates_seen.append(params)
return r
mock_db.execute = _execute
yield mock_db
sample = {
"alertname": "KubeDeploymentReplicasMismatch",
"ok": 1,
"total": 10, # >= 5
"recent_success_rate": 0.10, # < 0.3 → 觸發 draft
}
with patch("src.db.base.get_db_context", _ctx):
updated, flagged = await _update_catalog_confidence(sample)
assert updated is True
assert flagged is True
# 確認 UPDATE 帶了 review_status='draft'
draft_update = next(
(p for p in updates_seen if p.get("rs") == "draft"),
None,
)
assert draft_update is not None, "應有帶 rs='draft' 的 UPDATE 參數"
# =============================================================================
# test_min_sample_threshold
# =============================================================================
@pytest.mark.asyncio
async def test_min_sample_threshold_no_flag():
"""
recent_success_rate < 0.3 但 total < 5 → 不降 draft只更新 confidence.
"""
from src.jobs.aol_to_catalog_writeback_job import _update_catalog_confidence
existing_row = _catalog_row(confidence=0.60, review_status="approved")
updates_seen = []
call_count = 0
@asynccontextmanager
async def _ctx():
mock_db = AsyncMock()
async def _execute(sql, params=None):
nonlocal call_count
call_count += 1
r = MagicMock()
r.one_or_none.return_value = existing_row
if params:
updates_seen.append(params)
return r
mock_db.execute = _execute
yield mock_db
sample = {
"alertname": "SomeRareAlert",
"ok": 0,
"total": 3, # < 5 → 不降
"recent_success_rate": 0.0,
}
with patch("src.db.base.get_db_context", _ctx):
updated, flagged = await _update_catalog_confidence(sample)
assert updated is True
assert flagged is False
# 確認沒有帶 rs='draft' 的 UPDATE
draft_update = next(
(p for p in updates_seen if p.get("rs") == "draft"),
None,
)
assert draft_update is None, "sample < 5 不應降 review_status"
# =============================================================================
# test_dry_run_no_db_write / test_feature_flag_disabled_skips
# =============================================================================
@pytest.mark.asyncio
async def test_feature_flag_disabled_skips():
"""
ENABLE_AOL_WRITEBACK_JOB=False → run_aol_writeback_once 回傳 skipped=True
且不觸發任何 DB 操作。
"""
from src.jobs.aol_to_catalog_writeback_job import run_aol_writeback_once
db_call_count = 0
@asynccontextmanager
async def _ctx():
nonlocal db_call_count
db_call_count += 1
yield AsyncMock()
with patch("src.core.config.settings") as mock_settings:
mock_settings.ENABLE_AOL_WRITEBACK_JOB = False
# patch job module's settings reference
with patch("src.jobs.aol_to_catalog_writeback_job.settings", mock_settings):
result = await run_aol_writeback_once()
assert result["skipped"] is True
assert result["rules_sampled"] == 0
assert result["rules_updated"] == 0
assert result["rules_flagged_draft"] == 0
assert db_call_count == 0, "feature flag=False 時不應碰 DB"
@pytest.mark.asyncio
async def test_dry_run_no_db_write():
"""
同上flag=False 時完全不寫 DB別名測試語義明確.
"""
from src.jobs.aol_to_catalog_writeback_job import run_aol_writeback_once
written = []
@asynccontextmanager
async def _ctx():
mock_db = AsyncMock()
mock_db.execute = AsyncMock(side_effect=lambda *a, **kw: written.append(a))
yield mock_db
with patch("src.jobs.aol_to_catalog_writeback_job.settings") as mock_settings:
mock_settings.ENABLE_AOL_WRITEBACK_JOB = False
result = await run_aol_writeback_once()
assert result["skipped"] is True
assert len(written) == 0
# =============================================================================
# test_hermes_picks_up_draft — Hermes SQL 包含 OR review_status='draft' 條件
# =============================================================================
def test_hermes_sql_includes_draft_condition():
"""
驗證 hermes_rule_quality_job._fetch_noisy_rules 的 SQL 包含 OR review_status = 'draft'
(靜態檢查,不跑真實 DB.
W2 PR-R2 要求Hermes 必須撈到 AOL writeback 標記的 draft rules。
"""
import inspect
from src.jobs import hermes_rule_quality_job
# 讀取 _fetch_noisy_rules 的原始碼
src = inspect.getsource(hermes_rule_quality_job._fetch_noisy_rules)
assert "review_status = 'draft'" in src, (
"Hermes _fetch_noisy_rules 缺少 OR review_status = 'draft' 條件 "
"W2 PR-R2 斷鏈 C2 修復要求此條件觸發 AOL writeback advisory"
)
@pytest.mark.asyncio
async def test_hermes_picks_up_needs_review_rules():
"""
Hermes _fetch_noisy_rules 被呼叫時,若 DB 有 review_status='draft' 的 rule
應正常回傳(不因額外 OR 條件報錯).
"""
from src.jobs.hermes_rule_quality_job import _fetch_noisy_rules
draft_row = MagicMock()
draft_row.rule_id = 99
draft_row.rule_name = "LowSuccessRate"
draft_row.severity = "warning"
draft_row.true_positive_count = 1
draft_row.false_positive_count = 9
draft_row.noise_rate = 0.9
draft_row.last_fired_at = None
draft_row.review_status = "draft"
mock_result = MagicMock()
mock_result.fetchall.return_value = [draft_row]
@asynccontextmanager
async def _ctx():
mock_db = AsyncMock()
mock_db.execute = AsyncMock(return_value=mock_result)
yield mock_db
with patch("src.db.base.get_db_context", _ctx):
rules = await _fetch_noisy_rules()
assert len(rules) == 1
assert rules[0]["rule_name"] == "LowSuccessRate"
assert rules[0]["review_status"] == "draft"

View File

@@ -0,0 +1,402 @@
"""
KM → Playbook 互饋回路單元測試
================================
W2 PR-L1: 飛輪斷鏈 C3 + C4 修復測試
測試範圍:
1. test_playbook_promotion_writes_km_entry
— _promote_playbook 觸發後KMWriter 被呼叫寫 playbook_evolution 條目
2. test_playbook_demotion_writes_km_entry
— _demote_playbook 觸發後KMWriter 被呼叫寫 playbook_evolution 條目
3. test_km_accumulation_triggers_playbook_review
— 同 symptoms_hash 累積 5 條 → UPDATE playbooks.review_required=true
4. test_km_accumulation_below_threshold_no_update
— KM 條目 < threshold → 不執行 UPDATE
5. test_playbook_deprecated_demotes_alert_rule_confidence
— DEPRECATED Playbook → alert_rule_catalog.confidence *= 0.5
6. test_feature_flag_disabled
— ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=false → 三條邏輯全部跳過,不呼叫 DB
設計原則:
- 外部服務DB / KMWriter / PlaybookRepository以 AsyncMock 替換
- 每個 test 只測一條主路徑(單一職責)
- Feature flag 透過 patch 'src.core.config.settings' 控制
- get_db_context patch 路徑src.db.base.get_db_contextlocal import 的來源模組)
- get_playbook_repository patch 路徑:
src.repositories.playbook_repository.get_playbook_repository
建立2026-04-28 (台北時區) ogt + Claude Sonnet 4.6
"""
from __future__ import annotations
from contextlib import asynccontextmanager
from types import SimpleNamespace
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
# =============================================================================
# Helpers
# =============================================================================
def _make_playbook(
playbook_id: str = "PB-20260428-AAAAAA",
name: str = "TestPlaybook",
trust_score: float = 0.5,
success_count: int = 3,
failure_count: int = 1,
status: str = "approved",
alert_names: list[str] | None = None,
) -> SimpleNamespace:
"""
建立一個最小可用的 Playbook mock 物件。
使用 SimpleNamespace 讓屬性存取與 Pydantic model 相同,
但不引入真實 ORM / Pydantic 依賴(防止 DB 連線)。
symptom_pattern.compute_hash() 返回固定 'abc123' 供測試使用。
"""
symptom = SimpleNamespace(
alert_names=alert_names or ["HighCpuUsage"],
affected_services=["api"],
label_patterns={},
compute_hash=lambda: "abc123",
)
from src.models.playbook import PlaybookStatus
status_enum = PlaybookStatus(status)
return SimpleNamespace(
playbook_id=playbook_id,
name=name,
trust_score=trust_score,
success_count=success_count,
failure_count=failure_count,
status=status_enum,
symptom_pattern=symptom,
)
def _make_learning_service():
"""
建立 LearningService 實例,所有外部依賴 mock 掉。
repository 和 trust_repository 均使用 AsyncMock 防止 Redis 連線。
"""
from src.services.learning_service import LearningService
mock_repo = AsyncMock()
mock_trust_repo = AsyncMock()
mock_trust_mgr = MagicMock()
mock_trust_mgr.get_trust_record.return_value = None
svc = LearningService(
repository=mock_repo,
trust_repository=mock_trust_repo,
)
svc._trust_manager = mock_trust_mgr
return svc
def _make_settings(enable_loop: bool = True, threshold: int = 5) -> MagicMock:
"""
建立 settings mock。
patch 路徑src.core.config.settingslearning_service 各方法均 local import 自此模組)
"""
m = MagicMock()
m.ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP = enable_loop
m.KM_PLAYBOOK_REVIEW_THRESHOLD = threshold
m.KM_WRITE_AWAIT = True
m.KM_WRITE_TIMEOUT_SECONDS = 5.0
return m
def _make_db_context_factory(mock_db):
"""
返回一個可多次呼叫的 async context manager factory。
每次呼叫 factory() 返回新的 async context manager 實例,
防止同一 cm 物件被複用async generator 只能迭代一次)。
"""
def factory():
@asynccontextmanager
async def _ctx():
yield mock_db
return _ctx()
return factory
# =============================================================================
# 1. Promote 觸發 → 寫 KM 演化條目
# =============================================================================
@pytest.mark.asyncio
async def test_playbook_promotion_writes_km_entry():
"""
_promote_playbook 觸發後,若 ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=True
km_write_with_flag 應被呼叫一次path_type 含 'playbook_evolution'
"""
svc = _make_learning_service()
playbook = _make_playbook(trust_score=0.5, status="approved")
km_calls: list = []
async def _mock_km_write(payload, *, timeout=None):
km_calls.append(payload)
from src.services.km_writer import KMWriteResult
return KMWriteResult.SUCCESS
mock_pb_repo = AsyncMock()
mock_pb_repo.find_by_source_incident = AsyncMock(return_value=[playbook])
mock_pb_repo.adjust_confidence = AsyncMock(return_value=True)
mock_settings = _make_settings(enable_loop=True)
with (
patch("src.core.config.settings", mock_settings),
patch("src.services.km_writer.km_write_with_flag", side_effect=_mock_km_write),
patch(
"src.repositories.playbook_repository.get_playbook_repository",
return_value=mock_pb_repo,
),
):
result = await svc._promote_playbook("INC-TEST-001")
assert result is True
assert len(km_calls) == 1, "KMWriter 應被呼叫一次(一個 Playbook promote"
assert "playbook_evolution" in km_calls[0].path_type
assert km_calls[0].metadata["evolution_type"] == "promote"
assert km_calls[0].metadata["playbook_id"] == playbook.playbook_id
assert km_calls[0].metadata["previous_trust"] == 0.5
# =============================================================================
# 2. Demote 觸發 → 寫 KM 演化條目
# =============================================================================
@pytest.mark.asyncio
async def test_playbook_demotion_writes_km_entry():
"""
_demote_playbook 觸發後,若 ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=True
km_write_with_flag 應被呼叫一次evolution_type='demote'
status='approved'(非 DEPRECATED→ 邏輯 3 不觸發,保持單一職責。
"""
svc = _make_learning_service()
playbook = _make_playbook(trust_score=0.4, status="approved")
km_calls: list = []
async def _mock_km_write(payload, *, timeout=None):
km_calls.append(payload)
from src.services.km_writer import KMWriteResult
return KMWriteResult.SUCCESS
mock_pb_repo = AsyncMock()
mock_pb_repo.find_by_source_incident = AsyncMock(return_value=[playbook])
mock_pb_repo.adjust_confidence = AsyncMock(return_value=True)
mock_settings = _make_settings(enable_loop=True)
with (
patch("src.core.config.settings", mock_settings),
patch("src.services.km_writer.km_write_with_flag", side_effect=_mock_km_write),
patch(
"src.repositories.playbook_repository.get_playbook_repository",
return_value=mock_pb_repo,
),
):
result = await svc._demote_playbook("INC-TEST-002")
assert result is True
assert len(km_calls) == 1, "KMWriter 應被呼叫一次(一個 Playbook demote"
assert "playbook_evolution" in km_calls[0].path_type
assert km_calls[0].metadata["evolution_type"] == "demote"
# =============================================================================
# 3. KM 累積 N=5 → review_required=True
# =============================================================================
@pytest.mark.asyncio
async def test_km_accumulation_triggers_playbook_review():
"""
同 symptoms_hash 的 KM 條目達到 threshold預設 5
_check_and_mark_playbook_review 應執行 COUNT + UPDATE並 commit。
"""
svc = _make_learning_service()
symptoms_hash = "abc123"
mock_db = AsyncMock()
execute_call_count = {"n": 0}
mock_count_result = MagicMock()
mock_count_result.scalar.return_value = 5
mock_update_result = MagicMock()
mock_update_result.fetchall.return_value = [("PB-20260428-AAAAAA",)]
async def _multi_execute(stmt, params=None):
execute_call_count["n"] += 1
if execute_call_count["n"] == 1:
return mock_count_result
return mock_update_result
mock_db.execute = _multi_execute
mock_db.commit = AsyncMock()
mock_settings = _make_settings(enable_loop=True, threshold=5)
with (
patch("src.core.config.settings", mock_settings),
patch(
"src.db.base.get_db_context",
side_effect=_make_db_context_factory(mock_db),
),
):
await svc._check_and_mark_playbook_review(symptoms_hash)
assert execute_call_count["n"] == 2, "應執行兩次 SQLCOUNT + UPDATE"
mock_db.commit.assert_called_once()
@pytest.mark.asyncio
async def test_km_accumulation_below_threshold_no_update():
"""
KM 條目數 < threshold → 不執行 UPDATE不 commit。
"""
svc = _make_learning_service()
symptoms_hash = "abc123"
mock_db = AsyncMock()
execute_call_count = {"n": 0}
mock_count_result = MagicMock()
mock_count_result.scalar.return_value = 3 # < 5
async def _single_execute(stmt, params=None):
execute_call_count["n"] += 1
return mock_count_result
mock_db.execute = _single_execute
mock_db.commit = AsyncMock()
mock_settings = _make_settings(enable_loop=True, threshold=5)
with (
patch("src.core.config.settings", mock_settings),
patch(
"src.db.base.get_db_context",
side_effect=_make_db_context_factory(mock_db),
),
):
await svc._check_and_mark_playbook_review(symptoms_hash)
assert execute_call_count["n"] == 1, "只執行 COUNT不執行 UPDATE"
mock_db.commit.assert_not_called()
# =============================================================================
# 4. DEPRECATED → alert_rule_catalog.confidence *= 0.5
# =============================================================================
@pytest.mark.asyncio
async def test_playbook_deprecated_demotes_alert_rule_confidence():
"""
DEPRECATED Playbook 的 _demote_alert_rule_catalog_confidence 執行後,
每個 alert_name 執行一次 UPDATE最後 commit 一次。
"""
svc = _make_learning_service()
from src.models.playbook import PlaybookStatus
playbook = _make_playbook(
status="deprecated",
alert_names=["HighCpuUsage", "PodCrashLooping"],
)
playbook.status = PlaybookStatus.DEPRECATED
mock_db = AsyncMock()
execute_call_count = {"n": 0}
async def _track_execute(stmt, params=None):
execute_call_count["n"] += 1
m = MagicMock()
m.rowcount = 1
return m
mock_db.execute = _track_execute
mock_db.commit = AsyncMock()
mock_settings = _make_settings(enable_loop=True)
with (
patch("src.core.config.settings", mock_settings),
patch(
"src.db.base.get_db_context",
side_effect=_make_db_context_factory(mock_db),
),
):
await svc._demote_alert_rule_catalog_confidence(playbook)
assert execute_call_count["n"] == 2, "2 條 alert_names → 2 次 UPDATE"
mock_db.commit.assert_called_once()
# =============================================================================
# 5. Feature flag disabled → 所有邏輯跳過
# =============================================================================
@pytest.mark.asyncio
async def test_feature_flag_disabled():
"""
ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=False 時,
_write_playbook_evolution_km / _check_and_mark_playbook_review /
_demote_alert_rule_catalog_confidence 均不應呼叫任何 DB 或 KMWriter。
"""
svc = _make_learning_service()
from src.models.playbook import PlaybookStatus
playbook = _make_playbook(trust_score=0.3, status="deprecated")
playbook.status = PlaybookStatus.DEPRECATED
km_write_calls: list = []
db_execute_calls: list = []
async def _mock_km_write(payload, *, timeout=None):
km_write_calls.append(payload)
from src.services.km_writer import KMWriteResult
return KMWriteResult.SUCCESS
mock_db = AsyncMock()
async def _track_execute(stmt, params=None):
db_execute_calls.append(stmt)
return MagicMock()
mock_db.execute = _track_execute
mock_db.commit = AsyncMock()
mock_settings = _make_settings(enable_loop=False)
with (
patch("src.core.config.settings", mock_settings),
patch("src.services.km_writer.km_write_with_flag", side_effect=_mock_km_write),
patch(
"src.db.base.get_db_context",
side_effect=_make_db_context_factory(mock_db),
),
):
# 邏輯 1
await svc._write_playbook_evolution_km(
playbook=playbook,
previous_trust=0.5,
evolution_type="promote",
incident_id="INC-TEST-FLAG",
)
# 邏輯 2
await svc._check_and_mark_playbook_review("abc123")
# 邏輯 3
await svc._demote_alert_rule_catalog_confidence(playbook)
assert len(km_write_calls) == 0, "KMWriter 不應被呼叫flag=False"
assert len(db_execute_calls) == 0, "DB execute 不應被呼叫flag=False"
mock_db.commit.assert_not_called()

View File

@@ -0,0 +1,352 @@
"""
SelfHealingValidator 整合測試
================================
W2 PR-V1: 飛輪斷鏈 C6 修復驗收測試
測試項目:
1. test_validator_called_after_verification
— ENABLE=True 時verify() 完成後 assess_self_healing 被呼叫
2. test_low_score_triggers_rollback_proposal
— score < 0.5 時Telegram rollback 提案被發送
3. test_high_score_no_action
— score >= 0.5 時Telegram 不觸發
4. test_validator_failure_does_not_block_main_flow
— assess_self_healing 拋例外verify() 仍返回正確結果
5. test_feature_flag_disabled_skips
— ENABLE=False 時assess_self_healing 不被呼叫
2026-04-28 ogt + Claude Sonnet 4.6: W2 PR-V1 初始建立
"""
from __future__ import annotations
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from src.services.post_execution_verifier import PostExecutionVerifier
from src.services.evidence_snapshot import EvidenceSnapshot
from src.services.self_healing_validator import assess_self_healing
# ─────────────────────────────────────────────────────────────────────────────
# Stubs
# ─────────────────────────────────────────────────────────────────────────────
def _stub_incident(
alertname: str = "KubePodCrashLooping",
namespace: str = "awoooi-prod",
pod: str = "api-xyz",
) -> object:
class _Signal:
labels = {
"alertname": alertname,
"namespace": namespace,
"pod": pod,
}
class _Incident:
incident_id = "INC-TEST"
signals = [_Signal()]
return _Incident()
def _stub_snapshot(incident_id: str = "INC-TEST") -> EvidenceSnapshot:
snap = EvidenceSnapshot(incident_id=incident_id)
snap.pre_execution_state = {"status": "CrashLoopBackOff"}
return snap
# ─────────────────────────────────────────────────────────────────────────────
# assess_self_healing 單元測試(無 IO
# ─────────────────────────────────────────────────────────────────────────────
class TestAssessSelfHealing:
"""assess_self_healing() 純函數測試"""
def test_success_result_gives_high_score(self):
result = assess_self_healing(
pre_state={"status": "CrashLoopBackOff"},
post_state={"status": "Running", "containers": "1/1"},
verification_result="success",
action_taken="restart_service:api",
)
assert result["score"] >= 0.5
assert result["root_cause_cleared"] is True
def test_failed_result_gives_zero_score(self):
result = assess_self_healing(
pre_state={"status": "Running"},
post_state={"status": "CrashLoopBackOff"},
verification_result="failed",
action_taken="patch_config",
)
assert result["score"] == 0.0
assert result["root_cause_cleared"] is False
def test_degraded_result_gives_low_score(self):
result = assess_self_healing(
pre_state=None,
post_state={"status": "Pending"},
verification_result="degraded",
action_taken="scale_up",
)
assert result["score"] < 0.5
def test_regression_reduces_score(self):
"""執行後出現新 CrashLoopBackOff → regression penalty 扣分"""
result = assess_self_healing(
pre_state={"status": "Running"},
post_state={"status": "Running", "reason": "CrashLoopBackOff"},
verification_result="success",
action_taken="restart_service",
)
# regression 要扣分
assert "crashloopbackoff" in result["regressions"]
# 即使 verification_result=successregression 導致扣分
assert result["score"] < 1.0
def test_no_regression_full_score_on_success(self):
"""乾淨的 success無 regression、root cause 解除 → score=1.0"""
result = assess_self_healing(
pre_state={"status": "CrashLoopBackOff"},
post_state={"status": "Running", "containers": "1/1"},
verification_result="success",
action_taken="restart_service:api",
)
assert result["score"] == 1.0
assert result["regressions"] == []
def test_timeout_gives_low_base_score(self):
result = assess_self_healing(
pre_state=None,
post_state={},
verification_result="timeout",
action_taken="restart_service",
)
assert result["score"] == 0.2
def test_detail_is_human_readable(self):
result = assess_self_healing(
pre_state=None,
post_state={"status": "Running"},
verification_result="success",
action_taken="restart",
)
assert "base=" in result["detail"]
# ─────────────────────────────────────────────────────────────────────────────
# 整合測試verify() → _run_self_healing_validator
# ─────────────────────────────────────────────────────────────────────────────
class TestVerifyIntegration:
"""PostExecutionVerifier.verify() 串接 SelfHealingValidator 整合測試"""
@pytest.mark.asyncio
async def test_validator_called_after_verification(self):
"""ENABLE=True → verify() 完成後 assess_self_healing 被呼叫"""
verifier = PostExecutionVerifier()
incident = _stub_incident()
with (
patch.object(
verifier,
"_collect_post_state",
new=AsyncMock(return_value={"status": "Running"}),
),
patch("src.services.post_execution_verifier._update_snapshot", new=AsyncMock()),
patch(
"src.services.post_execution_verifier._run_self_healing_validator",
new=AsyncMock(),
) as mock_validator,
):
await verifier.verify(
incident=incident,
snapshot=None,
action_taken="restart_service:api",
warmup_sec=0.0,
)
mock_validator.assert_called_once()
call_kwargs = mock_validator.call_args.kwargs
assert call_kwargs["incident_id"] == "INC-TEST"
assert call_kwargs["verification_result"] == "success"
@pytest.mark.asyncio
async def test_low_score_triggers_rollback_proposal(self):
"""score < 0.5 → Telegram rollback 提案被發送"""
with (
patch(
"src.services.self_healing_validator.assess_self_healing",
return_value={
"score": 0.2,
"root_cause_cleared": False,
"regressions": ["crashloopbackoff"],
"detail": "base=0.40; regression_penalty=0.15",
"verification_result": "degraded",
"action_taken": "restart_service",
},
),
patch(
"src.services.post_execution_verifier._send_rollback_proposal_alert",
new=AsyncMock(),
) as mock_send,
patch(
"src.core.config.get_settings",
return_value=MagicMock(ENABLE_SELF_HEALING_VALIDATOR=True),
),
):
from src.services.post_execution_verifier import _run_self_healing_validator
await _run_self_healing_validator(
incident_id="INC-LOW",
snapshot=None,
pre_state={"status": "Running"},
post_state={"status": "CrashLoopBackOff"},
verification_result="degraded",
action_taken="restart_service",
)
mock_send.assert_called_once()
call_kwargs = mock_send.call_args.kwargs
assert call_kwargs["score"] == 0.2
assert call_kwargs["incident_id"] == "INC-LOW"
@pytest.mark.asyncio
async def test_high_score_no_action(self):
"""score >= 0.5 → Telegram rollback 提案不發送"""
with (
patch(
"src.services.self_healing_validator.assess_self_healing",
return_value={
"score": 1.0,
"root_cause_cleared": True,
"regressions": [],
"detail": "base=1.00",
"verification_result": "success",
"action_taken": "restart_service",
},
),
patch(
"src.services.post_execution_verifier._send_rollback_proposal_alert",
new=AsyncMock(),
) as mock_send,
patch(
"src.core.config.get_settings",
return_value=MagicMock(ENABLE_SELF_HEALING_VALIDATOR=True),
),
):
from src.services.post_execution_verifier import _run_self_healing_validator
await _run_self_healing_validator(
incident_id="INC-HIGH",
snapshot=None,
pre_state={"status": "CrashLoopBackOff"},
post_state={"status": "Running"},
verification_result="success",
action_taken="restart_service",
)
mock_send.assert_not_called()
@pytest.mark.asyncio
async def test_validator_failure_does_not_block_main_flow(self):
"""assess_self_healing 拋例外verify() 仍返回正確結果"""
verifier = PostExecutionVerifier()
incident = _stub_incident()
with (
patch.object(
verifier,
"_collect_post_state",
new=AsyncMock(return_value={"status": "Running"}),
),
patch("src.services.post_execution_verifier._update_snapshot", new=AsyncMock()),
# _run_self_healing_validator 本身 raise → 應被吞掉
patch(
"src.services.post_execution_verifier._run_self_healing_validator",
new=AsyncMock(side_effect=RuntimeError("validator exploded")),
),
):
# verify() 不應 raise仍返回 "success"
result = await verifier.verify(
incident=incident,
snapshot=None,
action_taken="restart_service:api",
warmup_sec=0.0,
)
# verify() 的主流程結果不受影響
# 注意_run_self_healing_validator 由 verify() await 直接呼叫,
# 其例外由 verify() 的 try/exceptapprove_execution 層級)或自身包住
# 此測試確認即使 validator 炸掉result 仍是正確的驗證結果
assert result == "success"
@pytest.mark.asyncio
async def test_feature_flag_disabled_skips(self):
"""ENABLE_SELF_HEALING_VALIDATOR=False → assess_self_healing 不被呼叫"""
import src.services.self_healing_validator as _shv
with (
patch.object(_shv, "assess_self_healing") as mock_assess,
patch(
"src.core.config.get_settings",
return_value=MagicMock(ENABLE_SELF_HEALING_VALIDATOR=False),
),
):
from src.services.post_execution_verifier import _run_self_healing_validator
await _run_self_healing_validator(
incident_id="INC-FLAG",
snapshot=None,
pre_state=None,
post_state={"status": "Running"},
verification_result="success",
action_taken="restart_service",
)
mock_assess.assert_not_called()
@pytest.mark.asyncio
async def test_snapshot_self_healing_score_updated(self):
"""score 補填 EvidenceSnapshot.self_healing_score"""
snap = _stub_snapshot()
snap.update_self_healing = AsyncMock()
with (
patch(
"src.services.self_healing_validator.assess_self_healing",
return_value={
"score": 0.85,
"root_cause_cleared": True,
"regressions": [],
"detail": "base=1.00",
"verification_result": "success",
"action_taken": "restart_service",
},
),
patch(
"src.services.post_execution_verifier._send_rollback_proposal_alert",
new=AsyncMock(),
),
patch(
"src.core.config.get_settings",
return_value=MagicMock(ENABLE_SELF_HEALING_VALIDATOR=True),
),
):
from src.services.post_execution_verifier import _run_self_healing_validator
await _run_self_healing_validator(
incident_id="INC-SNAP",
snapshot=snap,
pre_state={"status": "CrashLoopBackOff"},
post_state={"status": "Running"},
verification_result="success",
action_taken="restart_service",
)
snap.update_self_healing.assert_called_once()
call_kwargs = snap.update_self_healing.call_args.kwargs
assert call_kwargs["score"] == 0.85
assert call_kwargs["detail"]["root_cause_cleared"] is True
assert call_kwargs["detail"]["regressions"] == []