Your Name
|
b3a0f0d766
|
fix(telegram): dedup by fingerprint + 24h TTL to stop repeat alerts
CD Pipeline / tests (push) Successful in 2m22s
Code Review / ai-code-review (push) Successful in 57s
CD Pipeline / build-and-deploy (push) Successful in 21m3s
CD Pipeline / post-deploy-checks (push) Successful in 5m2s
Telegram 重複發告警鐵證(4 個 agent 真實數據):
- INC-6FE3BD (HostBackupFailed) 24h 內被推 15 次
- INC-FD6E21 (HostHighCpuLoad) 24h 內被推 6 次
- 06:44:18 同秒兩送 = pod 並發 race
根因:
1. `telegram_sent:{incident_id}` dedup key 綁 uuid4 隨機 INC ID,
同 fingerprint 換新 INC 完全不去重
2. dedup TTL=600s 比 incident_analysis_sweeper 重觸週期 1h、
alertmanager repeat_interval 4h 都短 → 每輪都過期通過
3. pod restart 走 _resend_unconfirmed_ready_tokens 用同一 incident_id key
→ 重啟必炸一波
修法(不消音、是「AI 認得這是同一事故」):
- decision_manager.py:207-225 dedup key 改 alertname+target fingerprint
- decision_manager.py:573-578 TTL 600s → 86400s (蓋住 sweeper 1h × alertmanager 4h)
- decision_manager.py:3189-3208 pod restart resend 路徑同步改 fingerprint
- incident_analysis_sweeper.py:37-42 sweeper_done TTL 3600s → 86400s
預期:同症狀 24h 內最多發 1 張 decision card;resolved 後 line 220-226
status check 會 early return,不影響復發偵測。
Tests: 35 passed (test_telegram_adr050 + test_decision_manager_docker_prune_routing)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-05-02 16:25:48 +08:00 |
|
Your Name
|
cc547736ab
|
feat(wave6-8): P2.1 fusion + P2.2 governance + P2.4 consensus + Wave 7/8 BLOCKER 修復
承接 Wave 6/7/8 多 engineer 在 agent 限額前完成的代碼,補 commit 解 production
HEAD 隱性 import error(decision_fusion 已被 decision_manager 引用但檔案 untracked)。
新增(後端核心):
- decision_fusion.py (562 行) — P2.1 方法 III(OpenClaw + Hermes + Elephant 三 LLM 融合)
- aiops_timeline.py + aiops_timeline_service.py — critic B4 修復
/api/v1/aiops/timeline endpoint,DB 存取抽到 service 層遵守 leWOOOgo 積木化
- migrations/p2_decision_fusion_columns.sql + rollback — approval_records fusion 欄位
修改(後端整合):
- decision_manager.py — fusion 三斷鏈修補(critic B1+B2+B3):
· B1: 寫 _evidence_snapshot_ref 到 token.proposal_data
· B2: fusion 前計算 complexity_score 並寫 token
· B3: fusion composite 寫 token.proposal_data["decision_fusion"]
- auto_approve.py — fusion + consensus 認識(critic B3+B5):
· composite > 0.7 → auto_execute_eligible bypass min_confidence
· source=consensus_engine + score>=0.6 → 規則可信路徑
- consensus_engine.py — db-fix _save_consensus 重用 agent_sessions
- governance_agent.py — db-fix _alert PG 寫入 ai_governance_events
- approval_db.py — fusion 3 欄位 + 2 partial index + CheckConstraint
- db/models.py — schema 對齊 migration
- core/config.py — vuln #1 修復:OLLAMA_URL/_FALLBACK_URL field_validator
拒絕公網 IP + 外部域名,僅允許私網/loopback/K8s SVC 白名單
- core/feature_flags.py — P2 fusion + consensus flags
- main.py — governance_agent lifespan 啟動
- failover_alerter.py — Wave8-X2: in-memory dedup fallback(Redis 拒絕後不 fail-open)
- ollama_*.py — metrics 整合 + recovery 改善
- auto_repair_service.py — verifier 接線
新增(測試 2438 行):
- test_decision_fusion.py / test_governance_agent.py / test_consensus_integration.py
- test_p2_db_fixes.py / test_wave8_fusion_fixes.py
- test_config_url_validation.py(vuln #1 12 tests)
- test_failover_alerter.py +Wave8-X2 in-memory dedup 補測
驗收: 116 tests pass (decision_fusion + wave8_fusion + config_url + consensus +
governance + p2_db_fixes + failover_alerter)
Conflict resolution:
- 3 檔(config.py + auto_approve.py + decision_manager.py)git stash pop 衝突
保留 stashed (engineer 最終版),補回 ValueError 「公網 IP」字樣對齊 test
Note: 此 commit 解 production HEAD 隱性 import error
仍未修: vuln #4 prompt injection / debugger B14 quota fail-closed
/ B25-B26 drain_pending_tasks / B8 governance fail alert
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Multiple Engineers (Wave 6/7/8) <noreply@anthropic.com>
|
2026-04-27 08:11:40 +08:00 |
|
OG T
|
9bfa6fc045
|
fix(sweeper): 限制只掃 48h 內 incident,防止歷史舊案洗版 Telegram
CD Pipeline / build-and-deploy (push) Has been cancelled
問題:
首次部署 sweeper 時,找到 117 個無 sweeper_done: 標記的舊 incident
(最舊 2026-04-09,7 天前) → 觸發全部 LLM 分析
舊 incident 資料格式 → OPENCLAW_NEMO timeout → Expert System 降級
confidence=0.2 "降級" → Telegram 連發相同格式告警洗版
修正:
加入 _MAX_INCIDENT_AGE_HOURS=48 過濾
只處理 48h 內的 INVESTIGATING incident
確保 created_at 時區安全(naive → UTC)
2026-04-16 Claude Sonnet 4.6 Asia/Taipei
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-16 01:27:02 +08:00 |
|
OG T
|
20b3fefca7
|
fix(sweeper): 修正 decision key 格式 BUG (decision:INC-* → sweeper_done:INC-*)
CD Pipeline / build-and-deploy (push) Has been cancelled
根本原因:
decision token 實際 key 格式為 decision:DEC-{HEX12}
sweeper 錯誤地查詢 decision:{incident_id} (永遠 = 0)
→ 每 90s 將 186 個 incident 全部列為「未分析」
→ 觸發大量重複 AI 分析請求 (雖 get_or_create_decision 有去重保護)
修正方式:
改用 sweeper_done:{incident_id} 輕量標記 (TTL 1h)
分析完成後才設標記,確保失敗的 incident 下輪仍會重試
get_or_create_decision 內部已有 COMPLETED/READY 去重,雙重保護
2026-04-16 Claude Sonnet 4.6 Asia/Taipei
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-16 01:20:16 +08:00 |
|
OG T
|
ce1a4d286e
|
feat(sweeper): 新增 Incident Analysis Sweeper — 自動觸發未分析 Incident AI 決策
Gap修復:
Signal Worker 創建 Incident 後,AI 分析只在 GET /api/v1/incidents 被呼叫時觸發
若前端無人訪問,新 Incident 永遠沒有 AI 分析與 Telegram 通知
解法:
新增 src/jobs/incident_analysis_sweeper.py
每 90 秒掃描無 decision token 的 INVESTIGATING incidents
自動背景觸發 get_or_create_decision() — Semaphore(3) 限流,每批最多 5 筆
main.py lifespan 啟動時 asyncio.create_task() 掛載
2026-04-16 Claude Sonnet 4.6 Asia/Taipei
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-16 01:08:30 +08:00 |
|