fix: 技術債清理 — report_generation 重試機制 + GAP-A4 文件化
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 15m46s
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 15m46s
技術債 #1: postmortem 發送失敗靜默吞掉 - 3 次指數退避重試 (2s → 4s → 6s) - 全失敗後送簡化降級通知到 SRE 群組 - 防止事後檢討默默消失 技術債 #2 (QueryBuilder 抽象): DEFER - 全專案僅 1 處用 outcome JSON path query - 違反「Don't design for hypothetical future requirements」 - 待第二 caller 出現再抽 技術債 #3 (E2E 測試): 已涵蓋 - test_gap_a4_placeholder_resolution.py TestMatchRuleRejection - Mission C prod 鏈路實測(KubePodCrashLooping) - Playwright K8s/Telegram staging 留待 staging 環境就緒 新增文件: - ADR-078-gap-a4-placeholder-resolution.md - LOGBOOK 2026-04-14 深夜收官條目 Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -460,23 +460,58 @@ class ReportGenerationService:
|
||||
resolved_at=resolved_at,
|
||||
)
|
||||
|
||||
# 技術債修復 (2026-04-14 Claude Sonnet 4.6): 3 次重試 + 指數退避
|
||||
# 失敗時發送告警到 SRE 群組,避免靜默吞掉錯誤
|
||||
import asyncio as _asyncio
|
||||
report_text = self.format_postmortem(data)
|
||||
from src.services.telegram_gateway import get_telegram_gateway
|
||||
gateway = get_telegram_gateway()
|
||||
|
||||
max_attempts = 3
|
||||
backoff_seconds = 2.0
|
||||
last_error: Exception | None = None
|
||||
for attempt in range(1, max_attempts + 1):
|
||||
try:
|
||||
await gateway.send_to_group(report_text, parse_mode="HTML")
|
||||
logger.info(
|
||||
"postmortem_sent",
|
||||
incident_id=incident_id,
|
||||
duration_minutes=duration_minutes,
|
||||
attempt=attempt,
|
||||
)
|
||||
return
|
||||
except Exception as e:
|
||||
last_error = e
|
||||
logger.warning(
|
||||
"postmortem_send_retry",
|
||||
incident_id=incident_id,
|
||||
attempt=attempt,
|
||||
max_attempts=max_attempts,
|
||||
error=str(e),
|
||||
)
|
||||
if attempt < max_attempts:
|
||||
await _asyncio.sleep(backoff_seconds * attempt)
|
||||
|
||||
# 3 次全失敗 → 記 error + 嘗試簡化降級通知(防止完全靜默)
|
||||
logger.error(
|
||||
"postmortem_failed",
|
||||
incident_id=incident_id,
|
||||
error=str(last_error),
|
||||
attempts=max_attempts,
|
||||
)
|
||||
try:
|
||||
report_text = self.format_postmortem(data)
|
||||
|
||||
from src.services.telegram_gateway import get_telegram_gateway
|
||||
gateway = get_telegram_gateway()
|
||||
await gateway.send_to_group(report_text, parse_mode="HTML")
|
||||
|
||||
logger.info(
|
||||
"postmortem_sent",
|
||||
incident_id=incident_id,
|
||||
duration_minutes=duration_minutes,
|
||||
fallback_text = (
|
||||
f"⚠️ <b>Postmortem 發送失敗 (3 次重試)</b>\n"
|
||||
f"Incident: <code>{incident_id}</code>\n"
|
||||
f"Duration: {duration_minutes:.1f} 分鐘\n"
|
||||
f"Error: {str(last_error)[:200]}"
|
||||
)
|
||||
except Exception as e:
|
||||
await gateway.send_to_group(fallback_text, parse_mode="HTML")
|
||||
except Exception as _fe:
|
||||
logger.error(
|
||||
"postmortem_failed",
|
||||
"postmortem_fallback_failed",
|
||||
incident_id=incident_id,
|
||||
error=str(e),
|
||||
error=str(_fe),
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user