fix(flywheel): 自動化飛輪六大能力修復(ADR-092 B3)
Some checks failed
run-migration / migrate (push) Failing after 22s
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 53s
Type Sync Check / check-type-sync (push) Successful in 2m54s
CD Pipeline / build-and-deploy (push) Has been cancelled
Ansible Lint / lint (push) Has been cancelled
Some checks failed
run-migration / migrate (push) Failing after 22s
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 53s
Type Sync Check / check-type-sync (push) Successful in 2m54s
CD Pipeline / build-and-deploy (push) Has been cancelled
Ansible Lint / lint (push) Has been cancelled
【根因鏈修復】 MCP Provider bugs → PreDecisionInvestigator 失敗 → Agent Debate 無上下文 → LLM 逾時 → description="待分析" → ADR-091 鐵閘攔截 → tg_sent 未設 → W-2 Watchdog 誤報「靜默故障」 【六大修復】 1. MCP Provider 三蟲修復 - ssh_provider: asyncssh.run() → conn.run() - prometheus_provider: KeyError 'query' → .get() 容錯 - k8s_provider: 空 pod_name → 早返回錯誤字典 2. Agent Debate / 決策品質 - decision_manager: 逾時降級文字改為明確描述(繞過 ADR-091 鐵閘) - intent_classifier: LLM 逾時降級至關鍵字分類(非 None) 3. Watchdog 誤報修復(ADR-092 B3) - W-2: tg_sent Redis TTL → telegram_message_id IS NULL(DB 真值) - W-5 新增: suggested_action IN 空/待分析/NO_ACTION + tg_id IS NULL - approval_timeout_resolver: 60min → 15min,batch 50 → 200 4. Config Drift 自動化 - drift_adopt_service: auto_adopt_if_safe() 六條件安全閘 - drift.py: 背景任務先嘗試自動採納再發人工 Telegram 卡片 5. Playbook 飛輪穩定 - playbook_seed_service: 修復幂等性(deprecated 不視為缺失) - playbook_evolver: 只載 DRAFT+APPROVED(非全部 294 筆) 6. 可觀測性 - alert_rule_engine: auto_rule 結構化日誌 + Redis 計數器(pipeline) - auto_approve: reject 原因 Redis 計數器 - heartbeat_report_service: 新增「⚙️ 自動化統計(今日)」區塊 【待人工執行】 psql $DATABASE_URL -f apps/api/migrations/cleanup_duplicate_deprecated_playbooks.sql Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
163
docs/LOGBOOK.md
163
docs/LOGBOOK.md
@@ -6,6 +6,142 @@
|
||||
|
||||
---
|
||||
|
||||
## 📍 2026-04-24 — 12-Agent 全景盤點 + 六大自動化飛輪修復
|
||||
|
||||
### 根因(截圖告警分析)
|
||||
- META 告警(W-2)誤判:tg_sent: Redis TTL 24h 過期被誤報為「Telegram 靜默」,實際 Telegram 早已發送
|
||||
- 真正根因:MCP Provider 三個 Bug → Agent Debate 90s 超時 → description="待分析" → ADR-091 鐵閘不推 Telegram
|
||||
- Config Drift 全人工:無自動採納觸發鏈路
|
||||
- Playbook Evolver 迴圈:HTTP 5xx 重複建立/deprecated(294 deprecated / 25 approved)
|
||||
|
||||
### 修復內容(706行+,17 檔)
|
||||
|
||||
| 模組 | 修復 |
|
||||
|------|------|
|
||||
| `ssh_provider.py` | `asyncssh.run` → `conn.run()`(API 用錯) |
|
||||
| `prometheus_provider.py` | KeyError 'query' → `.get()` fallback + `promql` alias |
|
||||
| `k8s_provider.py` | 空 pod_name → early return error dict |
|
||||
| `ai_slo_watchdog_job.py` | W-2 改用 `telegram_message_id IS NULL`;W-5 新增(Agent Debate 卡住) |
|
||||
| `approval_timeout_resolver.py` | 1h → 15min;BATCH_LIMIT 50 → 200 |
|
||||
| `approval_db.py` | tg_sent TTL 24h → 30h(buffer 防邊緣誤判) |
|
||||
| `drift_adopt_service.py` | 新增 `auto_adopt_if_safe()`(6 條件自動採納 PR) |
|
||||
| `drift.py` | 背景任務加自動採納邏輯(低風險走 auto,高風險走人工) |
|
||||
| `playbook_seed_service.py` | 冪等 SQL 修復(去掉 `AND status != 'deprecated'` 防重複建立) |
|
||||
| `playbook_evolver.py` | `_fetch_all_active_playbooks` 只載 APPROVED+DRAFT,不載 deprecated |
|
||||
| `alert_rule_engine.py` | 自動規則生成加 telemetry + Redis pipeline 原子 incr/expire |
|
||||
| `auto_approve.py` | 拒絕原因 Redis 計數(供系統報告展示) |
|
||||
| `heartbeat_report_service.py` | 新增「自動化統計」區塊(今日規則/KM/Drift/Playbook) |
|
||||
| `decision_manager.py` | Agent Debate 超時降級文字(通過 ADR-091 鐵閘) |
|
||||
| `intent_classifier.py` | LLM 超時 → keyword fallback(不浪費 5s 等待) |
|
||||
| `migrations/cleanup_duplicate_deprecated_playbooks.sql` | 一次性清理 294 筆重複 deprecated |
|
||||
|
||||
### Critic 審查後追加修復
|
||||
- `heartbeat_report_service.py` SQL:移除 `AT TIME ZONE`(timestamptz 直接比較);drift_today 改查 drift_reports 表
|
||||
- `drift_adopt_service.py` 雙重 Telegram 問題:`suppress_notification=True` 避免 auto_adopt 重複發
|
||||
- `alert_rule_engine.py` Redis race:pipeline 原子化 incr+expire
|
||||
- `ai_slo_watchdog_job.py` W-5:改用 `action IS NULL/空 + telegram_message_id IS NULL` 更可靠
|
||||
|
||||
### 待手動執行
|
||||
```bash
|
||||
psql $DATABASE_URL -f apps/api/migrations/cleanup_duplicate_deprecated_playbooks.sql
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📍 2026-04-22 — 系統報告動態化:新增 5 大區塊(commit 9244c5e)
|
||||
|
||||
### 需求
|
||||
統帥:「系統報告需要更全面、更完整,服務增加了很多,必須動態滾動增刪」
|
||||
|
||||
### 實作
|
||||
| 新區塊 | 資料來源 | 說明 |
|
||||
|--------|---------|------|
|
||||
| 📊 告警流水線(24h) | `approval_records.status` | total/pending/success/failed |
|
||||
| 🗄️ DB & Redis | PG `SELECT 1` + Redis info | 連線狀態 + key 數 |
|
||||
| ☸️ K8s Pods | kubectl get pods | ready/restart count |
|
||||
| ⏱️ Scanner 狀態 | Redis daily lock TTL | 今日是否已執行 |
|
||||
| 🤖 Telegram Bot | Redis `telegram:polling_leader` | polling leader 是否存活 |
|
||||
|
||||
- 5 個 probe 方法用 `asyncio.gather(return_exceptions=True)` 並行,任一失敗不影響其他
|
||||
- `_build_warnings()` 新增 DB/Redis/PENDING>10/Pod 未就緒 四種警告條件
|
||||
- 新增 5 個 dataclass:AlertPipelineStats / DbRedisStats / PodInfo / ScannerStats / TelegramBotStats
|
||||
|
||||
### Commit
|
||||
- `9244c5e` feat(heartbeat): 系統報告新增 5 大動態區塊
|
||||
|
||||
---
|
||||
|
||||
## 📍 2026-04-22 早 — 日報重發 + 自動修復 0% 兩大根因修復(commits ef1353b + 88af639)
|
||||
|
||||
### 問題
|
||||
統帥:「日報重複發送兩次」+「自動修復成功率 0.0%」
|
||||
|
||||
### 根因
|
||||
|
||||
| 問題 | 根因 | 修法 |
|
||||
|------|------|------|
|
||||
| 日報重發 | `run_daily_report_loop` 沒有呼叫 `try_acquire_daily_lock`(其他 3 個 scanner 都有),4 個 Pod 各自送一份 | sleep 後加 `try_acquire_daily_lock("daily_report")`,搶不到的 Pod 直接 `continue` |
|
||||
| 修復率 0% | `_collect_repair_stats` 查 `incidents.outcome->>'execution_success'`,但整條執行鏈路從未將 `execution_success` 寫入此 JSON 欄位 | 改查 `approval_records.status = 'EXECUTION_SUCCESS/FAILED'`(唯一可靠 source of truth) |
|
||||
| SQL 大小寫 | DB 以 SQLEnum 儲存 enum name(EXECUTION_FAILED 大寫),SQL 用小寫比對 | 加 `UPPER(status::text)` 保證命中 |
|
||||
|
||||
### 驗證
|
||||
- Live DB 驗證:修正後 SQL → `success=0, failed=2`(之前永遠 0/0)
|
||||
- 明早 08:00 報告應顯示真實成功率(今日 0/2 = 0%,但數字正確)
|
||||
|
||||
### Commits
|
||||
- `ef1353b` 主修復(leader lock + 改查 approval_records)
|
||||
- `88af639` SQL 大小寫修正
|
||||
|
||||
---
|
||||
|
||||
## 📍 2026-04-22 凌晨 — Telegram 按鈕靜默兩大根因修復(commit 1625e7b)
|
||||
|
||||
### 問題
|
||||
統帥:「按鈕按下去,會產生的所有操作和結果,也都沒有回覆到 Telegram 群組上!」
|
||||
|
||||
### 根因全景(Debugger 全景調查)
|
||||
|
||||
| 陷阱 | 症狀 | 根因 | 修法 |
|
||||
|------|------|------|------|
|
||||
| T1 容量預測按鈕靜默 | "已處理"/"忽略 24h" 按下後群組無回覆 | `_handle_ai_advisory_action` 只呼叫 `_answer_callback`(toast 2-3 秒消失),從未 `sendMessage` 到群組 | 加 `message_id` 參數,toast 後發 `sendMessage reply` 到群組 |
|
||||
| T2 已解決告警批准靜默 | 再按「批准」→ 出現「⚡ 執行中...」但永遠沒結果 | `sign_approval` early-return(status != pending),但代碼仍呼叫 `_notify_approval_result` 發「執行中...」;`execute_approved_action` 因 status != APPROVED 跳過 → 永無結果 | 僅 `approval.status == APPROVED` 才發「執行中...」;否則發「ℹ️ 此告警已處理(狀態:...)」 |
|
||||
|
||||
### 修復範圍
|
||||
- `telegram_gateway.py:_handle_ai_advisory_action` — 加 `message_id` + 群組 reply
|
||||
- `telegram_gateway.py:_execute_approval_action` — 非 PENDING 狀態正確通知
|
||||
|
||||
### 部署
|
||||
- Commit: `1625e7b`(push gitea main)
|
||||
- Gitea pipeline build → Harbor push → K8s 滾動更新 → 02:13 完成
|
||||
- 新 image: `1625e7bd19017d9287fef55a5660ac125a413626`
|
||||
|
||||
---
|
||||
|
||||
## 📍 2026-04-21 晚 — 全流程三斷點修復(commit 4fc1f49)
|
||||
|
||||
### 根因全景
|
||||
| 斷點 | 症狀 | 根因 | 修法 |
|
||||
|------|------|------|------|
|
||||
| D1 飛輪 SLO 公式 | 執行成功率永遠 0.0% | `execution_count` 欄位不存在於 Playbook 模型 → `total_exec` 恆為 0 | `flywheel_stats_service.py` 改讀 `success_count + failure_count` |
|
||||
| D2 幻覺降級風險未清 | NO_ACTION 仍等 TG 批准 | `_validate_deployment_inventory` 降級 NO_ACTION 後 `risk_level` 未重置 → 原 HIGH/CRITICAL 風險觸發 PENDING | `openclaw.py` 加 `result.risk_level = AIRiskLevel.LOW` |
|
||||
| D3 NO_ACTION PENDING 積壓 | 非破壞性動作等人工批准 | `webhooks.py` 未依 suggested_action 調整 risk_level → INVESTIGATE/OBSERVE/NO_ACTION 全走 Telegram | 兩處 alert path 加 `_non_destructive_actions` LOW risk 強制 |
|
||||
|
||||
### 修復效果
|
||||
- 新告警若 LLM 返回 NO_ACTION/INVESTIGATE/OBSERVE → 立即 LOW risk → auto-approve → `approval_execution.py` NO_ACTION handler → EXECUTION_SUCCESS
|
||||
- `_validate_deployment_inventory` 幻覺降級後,後續批准路徑完全跳過 Telegram
|
||||
- 飛輪 SLO 指標有實際執行後將反映真實數字(有執行才有分母)
|
||||
|
||||
### 待執行(Pod 更新後)
|
||||
```bash
|
||||
# 清理 35+ 筆舊 PENDING(無 tg_msg 且超 2h)
|
||||
kubectl -n awoooi-prod exec $POD -- python -c "..." # 見 LOGBOOK 說明
|
||||
```
|
||||
|
||||
### Commit
|
||||
- `4fc1f49` (Gitea pipeline 部署中)
|
||||
|
||||
---
|
||||
|
||||
## 📍 2026-04-21 下午 — BUTTON_DATA_INVALID 根治 + Gitea Code Review 修復
|
||||
|
||||
### 問題
|
||||
@@ -1294,3 +1430,30 @@ CR 修補:
|
||||
2. 重啟後 → 被封存 yaml_rule playbooks 復活(C2 seeder 修復)
|
||||
3. AI 自動生成新規則 → 立即出現對應 APPROVED Playbook(C3 接線)
|
||||
4. Watchdog W-4 → APPROVED 數量為 0 時 TYPE-8M 告警(C4 感知)
|
||||
|
||||
---
|
||||
|
||||
## 2026-04-24(台北)— Playbook 重複建立/封存迴圈根治
|
||||
|
||||
**觸發**:DB 查詢顯示 `HTTP 5xx 錯誤率過高` Playbook 建立 7 次以上,294 筆 deprecated / 25 筆 approved。
|
||||
|
||||
### 根因
|
||||
|
||||
C1(evolver 加 YAML_RULE guard)+ C2(seeder SQL `AND status != 'deprecated'`)邏輯衝突:
|
||||
- C1 已讓 evolver 不再封存 YAML_RULE → deprecated 記錄只剩 C1 上線前的歷史垃圾
|
||||
- C2 讓 seeder「看到 deprecated 就重建」→ 歷史垃圾永遠在 DB → 每次重啟建一筆新 Playbook
|
||||
- C1 保護新建的 Playbook 不被封存,但 deprecated 歷史記錄不清除 → 無限重建
|
||||
|
||||
### 修復
|
||||
|
||||
| 檔案 | 修改 |
|
||||
|------|------|
|
||||
| `playbook_seed_service.py` | 移除 `AND status != 'deprecated'`,同名 yaml_rule 任何 status 都視為已存在 |
|
||||
| `playbook_evolver.py` | `_fetch_all_active_playbooks` 改分兩次 status 查詢(DRAFT + APPROVED),不載入 deprecated |
|
||||
| `migrations/cleanup_duplicate_deprecated_playbooks.sql` | 每個 name 保留最新一筆 deprecated,刪除其餘(需手動套用一次) |
|
||||
|
||||
### 待手動執行
|
||||
|
||||
```bash
|
||||
psql $DATABASE_URL -f apps/api/migrations/cleanup_duplicate_deprecated_playbooks.sql
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user