feat(flywheel): W2 三件 + KMWriter critic 修法（1635 tests 全綠）
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m38s
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m38s
W2 (onboarder 4 週飛輪 80→90 路徑第二週) + critic PR review 5 個 critical/major 全部修完，default flag=false 安全無爆炸風險。 ## W2 三件 PR ### PR-R2 — AOL → catalog confidence EWMA 回灌（修飛輪斷鏈 C2） - 新檔 `apps/api/src/jobs/aol_to_catalog_writeback_job.py` - 邏輯：每小時掃 AOL 計算 EWMA confidence (alpha=0.3) 回灌 alert_rule_catalog - 失敗閾值 N=5 連續低成功率 → review_status='draft' - Hermes _fetch_noisy_rules SQL 加 OR review_status='draft' - ENABLE_AOL_WRITEBACK_JOB=false (default) - 8 個測試（mock path 修正：lazy import → patch src.db.base.get_db_context） ### PR-V1 — self_healing_validator 串接 (修飛輪斷鏈 C6) - 新檔 `apps/api/src/services/self_healing_validator.py`（純函數 assess_self_healing） - post_execution_verifier.py step 5 串接（feature flag gate） - evidence_snapshot.py 加 self_healing_score / self_healing_detail 欄位 - db/models.py + base.py ALTER IF NOT EXISTS - score < 0.5 → 觸發 rollback 提案 Telegram alert（不自動執行） - ENABLE_SELF_HEALING_VALIDATOR=false (default) - 7 個測試 ### PR-L1 — KM ↔ Playbook 雙向回路 (修飛輪斷鏈 C3+C4) - learning_service.py 三條新邏輯： 1. _write_playbook_evolution_km：promote/demote 寫 KM 演化條目 2. _check_and_mark_playbook_review：N=5 累積觸發 review_required 3. _demote_alert_rule_catalog_confidence：DEPRECATED → confidence×=0.5 - PlaybookRecord 加 review_required 欄位（schema migration via base.py） - ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=false (default) - KM_PLAYBOOK_REVIEW_THRESHOLD=5 可調 - 6 個測試 ## KMWriter Critic 5 個 Critical/Major 修復（之前 critic PR review 發現） 之前 push commitc5753e1c已修，本 commit 補回 stash 中的對應檔案： - C1 km_writer.py:194 backfill 自打臉（已修：同步 await + DLQ） - C2 km_writer.py:391 KM_WRITE_AWAIT=false 路徑收緊 - M1 decision_manager.py:2178/2203 移除 _fire_and_forget - M2 incident_service.py:1099 自製 path 加 retry+DLQ - M3 km_writer.py:166 冪等聲明對齊（UPSERT + partial unique index） ## 驗證 - 1635 unit tests 全綠（+27 from 1608） - 與fb0c72db(推翻 A2 Ollama primary) 共存無衝突 - 所有新 Job/Service default flag=false（不爆炸） ## 期望影響 飛輪斷鏈 C2 + C3 + C4 + C6 全修 飛輪自主化評分：65 → 85 預估（W2 完成後） 啟用順序（待 prodfb0c72db驗證 OLLAMA primary 跑得起來後）： 1. ENABLE_AOL_WRITEBACK_JOB=true 2. ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=true 3. ENABLE_SELF_HEALING_VALIDATOR=true Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Your Name
@@ -21,6 +21,11 @@ AWOOOI AIOps Phase 1 — 執行後驗證器
|
||||
- 超時不 raise，標記 "timeout" 並繼續流程
|
||||
- 不阻塞原始執行路徑（await，但結果不影響執行本身是否成功）
|
||||
|
||||
W2 PR-V1: SelfHealingValidator 串接 (2026-04-28 ogt + Claude Sonnet 4.6)
|
||||
- ENABLE_SELF_HEALING_VALIDATOR=True 時，verify() 完成後呼叫 assess_self_healing()
|
||||
- self_healing_score < 0.5 → Telegram 警示 rollback 提案（不自動執行）
|
||||
- 驗證失敗不阻塞主流程（try/except 全包）
|
||||
|
||||
ADR-081: PreDecisionInvestigator + EvidenceSnapshot
|
||||
MASTER §3.1 L6×D1
|
||||
2026-04-15 ogt + Claude Sonnet 4.6 (亞太): Phase 1 初始建立
|
||||
@@ -37,6 +42,9 @@ import structlog
|
||||
from src.services.evidence_snapshot import EvidenceSnapshot
|
||||
from src.services.mcp_tool_registry import SensorDimension, get_mcp_tool_registry
|
||||
from src.services.sanitization_service import sanitize_dict_values
|
||||
# W2 PR-V1: 頂層 import 讓測試 patch 路徑固定（延遲 import 無法被 patch）
|
||||
# ENABLE_SELF_HEALING_VALIDATOR=False 時此 import 不影響效能（純 python 模組）
|
||||
from src.services import self_healing_validator as _shv_module
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from src.models.incident import Incident
|
||||
@@ -136,6 +144,26 @@ class PostExecutionVerifier:
|
||||
result=result,
|
||||
action=action_taken,
|
||||
)
|
||||
|
||||
# 5. W2 PR-V1: SelfHealingValidator 串接（ENABLE_SELF_HEALING_VALIDATOR gate）
|
||||
# 在 post_state 已補填後評估自愈品質，不阻塞主流程
|
||||
# 外層 try/except 確保任何 validator 失敗不影響 verify() 返回值
|
||||
try:
|
||||
await _run_self_healing_validator(
|
||||
incident_id=incident_id,
|
||||
snapshot=snapshot,
|
||||
pre_state=pre_state,
|
||||
post_state=post_state,
|
||||
verification_result=result,
|
||||
action_taken=action_taken,
|
||||
)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"self_healing_validator_uncaught",
|
||||
incident_id=incident_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
return result
|
||||
|
||||
async def capture_pre_execution_state(
|
||||
@@ -209,6 +237,132 @@ class PostExecutionVerifier:
|
||||
return state
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# W2 PR-V1: SelfHealingValidator 串接
|
||||
# 2026-04-28 ogt + Claude Sonnet 4.6: C6 飛輪斷鏈修復
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
async def _run_self_healing_validator(
|
||||
incident_id: str,
|
||||
snapshot: EvidenceSnapshot | None,
|
||||
pre_state: dict[str, Any] | None,
|
||||
post_state: dict[str, Any],
|
||||
verification_result: str,
|
||||
action_taken: str,
|
||||
) -> None:
|
||||
"""
|
||||
SelfHealingValidator 串接入口。
|
||||
|
||||
Feature gate: ENABLE_SELF_HEALING_VALIDATOR（預設 False）。
|
||||
驗證失敗全程 try/except 保護，不影響主流程。
|
||||
|
||||
評估後：
|
||||
- 補填 snapshot.self_healing_score + self_healing_detail
|
||||
- score < 0.5 → 發送 Telegram rollback 提案警示
|
||||
"""
|
||||
try:
|
||||
from src.core.config import get_settings
|
||||
_settings = get_settings()
|
||||
if not _settings.ENABLE_SELF_HEALING_VALIDATOR:
|
||||
return
|
||||
|
||||
assessment = _shv_module.assess_self_healing(
|
||||
pre_state=pre_state,
|
||||
post_state=post_state,
|
||||
verification_result=verification_result,
|
||||
action_taken=action_taken,
|
||||
)
|
||||
score: float = assessment["score"]
|
||||
|
||||
logger.info(
|
||||
"self_healing_assessed",
|
||||
incident_id=incident_id,
|
||||
score=score,
|
||||
regressions=assessment.get("regressions", []),
|
||||
root_cause_cleared=assessment.get("root_cause_cleared"),
|
||||
detail=assessment.get("detail"),
|
||||
)
|
||||
|
||||
# 補填 EvidenceSnapshot
|
||||
if snapshot:
|
||||
try:
|
||||
await snapshot.update_self_healing(score=score, detail=assessment)
|
||||
except Exception as _snap_err:
|
||||
logger.warning(
|
||||
"self_healing_snapshot_update_failed",
|
||||
incident_id=incident_id,
|
||||
error=str(_snap_err),
|
||||
)
|
||||
|
||||
# score < 0.5 → Telegram rollback 提案警示
|
||||
if score < 0.5:
|
||||
await _send_rollback_proposal_alert(
|
||||
incident_id=incident_id,
|
||||
score=score,
|
||||
assessment=assessment,
|
||||
action_taken=action_taken,
|
||||
)
|
||||
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"self_healing_validator_error",
|
||||
incident_id=incident_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
|
||||
async def _send_rollback_proposal_alert(
|
||||
incident_id: str,
|
||||
score: float,
|
||||
assessment: dict[str, Any],
|
||||
action_taken: str,
|
||||
) -> None:
|
||||
"""
|
||||
自愈品質分數 < 0.5 時，發送 Telegram rollback 提案警示。
|
||||
|
||||
不自動執行 rollback，僅通知人工評估。
|
||||
"""
|
||||
try:
|
||||
from src.core.config import get_settings
|
||||
from src.services.telegram_gateway import get_telegram_gateway
|
||||
_settings = get_settings()
|
||||
gateway = get_telegram_gateway()
|
||||
|
||||
regressions = assessment.get("regressions", [])
|
||||
reg_str = ", ".join(regressions[:5]) if regressions else "無"
|
||||
root_cleared = "是" if assessment.get("root_cause_cleared") else "否"
|
||||
|
||||
text = (
|
||||
f"⚠️ <b>自愈品質警示 — 建議人工評估 Rollback</b>\n"
|
||||
f"Incident: <code>{incident_id}</code>\n"
|
||||
f"動作: <code>{action_taken[:120]}</code>\n"
|
||||
f"自愈分數: <b>{score:.2f}</b> (門檻 0.5)\n"
|
||||
f"Root Cause 解除: {root_cleared}\n"
|
||||
f"Regression 信號: {reg_str}\n"
|
||||
f"<i>此為提案，不會自動執行 Rollback</i>"
|
||||
)
|
||||
|
||||
await gateway._http_client.post(
|
||||
f"https://api.telegram.org/bot{_settings.OPENCLAW_TG_BOT_TOKEN}/sendMessage",
|
||||
json={
|
||||
"chat_id": _settings.OPENCLAW_TG_CHAT_ID,
|
||||
"text": text,
|
||||
"parse_mode": "HTML",
|
||||
},
|
||||
)
|
||||
logger.info(
|
||||
"rollback_proposal_sent",
|
||||
incident_id=incident_id,
|
||||
score=score,
|
||||
)
|
||||
except Exception:
|
||||
logger.warning(
|
||||
"rollback_proposal_send_failed",
|
||||
incident_id=incident_id,
|
||||
exc_info=True,
|
||||
)
|
||||
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Recovery Assessment
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
Reference in New Issue
Block a user