feat(api): ADR-038 Circuit Breaker 整合 + Graceful Degradation

sentry_webhook.py:
- 整合 OpenClawGuard (Circuit Breaker + Semaphore)
- 斷路狀態快速失敗,不呼叫 OpenClaw
- 並發控制: 最多 3 個同時 LLM 推理

anomaly_counter.py:
- record_anomaly() Redis 故障 Graceful Degradation
- 失敗時返回預設 AnomalyFrequency (count=0)
- 不中斷主流程

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
OG T
2026-03-29 15:55:51 +08:00
parent 39396dc57a
commit 89a2339796
2 changed files with 83 additions and 23 deletions

View File

@@ -132,14 +132,54 @@ class AnomalyCounter:
"""
記錄一次異常發生
ADR-038/039: Redis 故障時 Graceful Degradation
- 記錄錯誤但不中斷主流程
- 返回空的 AnomalyFrequency頻率計數 = 0
Args:
anomaly_signature: 異常簽名字典
Returns:
AnomalyFrequency: 當前頻率統計
AnomalyFrequency: 當前頻率統計Redis 失敗時返回預設值)
"""
anomaly_key = self.hash_signature(anomaly_signature)
now = datetime.now()
try:
return await self._record_anomaly_impl(
anomaly_key=anomaly_key,
anomaly_signature=anomaly_signature,
now=now,
)
except Exception as e:
# ADR-038: Redis 故障 Graceful Degradation
logger.warning(
"anomaly_counter_redis_error",
error=str(e),
anomaly_key=anomaly_key,
fallback="returning_default_frequency",
)
# 返回預設值,不阻擋主流程
return AnomalyFrequency(
anomaly_key=anomaly_key,
count_1h=0,
count_24h=0,
count_7d=0,
count_30d=0,
first_seen=now,
last_seen=now,
auto_repair_count=0,
permanent_fix_applied=False,
escalation_level=None,
)
async def _record_anomaly_impl(
self,
anomaly_key: str,
anomaly_signature: dict,
now: datetime,
) -> AnomalyFrequency:
"""實際的異常記錄邏輯(可能拋出 Redis 異常)"""
timestamp = now.timestamp()
timeline_key = f"{self.PREFIX_TIMELINE}{anomaly_key}"