wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity
1,287 Commits 199 Branches 2 Tags
2cef2098d3e24389c67e7a7e36710b487192b36d
Commit Graph

60 Commits

Author SHA1 Message Date
OG T
2cef2098d3 feat(adr-075): 修復 Telegram 動態按鈕 4 個斷點 + 新增 7 種告警分類
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
斷點 A: decision_manager 提取 alert_category/notification_type 傳入 send_approval_card
斷點 B: send_approval_card 新增參數並傳遞至 _build_inline_keyboard
斷點 C: 互動型通知 (TYPE-3/4/4D/8M) 禁止發 SRE 群組，防 nonce 洩漏
斷點 D: _CATEGORY_BUTTONS k8s_workload → kubernetes + 新增 6 類按鈕組

classify_alert_early 新增: alertchain_health, flywheel_health, storage,
devops_tool, external_site, ssl_cert, host_resource (從 infrastructure 分離)
Test: 52 classify + 664 total passed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 18:35:56 +08:00
OG T
c09521a1c6 fix(cr): Code Review P0/P1 全修補 — 積木化+SSH路由+安全守衛順序
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m30s
Details 
P0-1: classify_alert_early 移至 incident_service (Service層)，webhooks.py import 修正
P0-2: _ssh_execute() 改用 self._ssh，移除冗餘 SSHProvider() 實例化
P1-1: infrastructure SSH routing 移至 kubectl safety guard 之前，docker指令不再被攔截
P1-2: alert_rule_engine 新增 get_risk_for_alertname() public API
P1-3: classify_notification() docstring 修正 ORM→Pydantic

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:51:12 +08:00
OG T
dbc77c5e62 feat(flywheel): Phase 3 — decision_manager Tier 3 七大修復 (首席架構師授權)
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
ADR-073 Phase 3 全部完成:

3-1: TYPE-1 triage guard
- get_or_create_decision() 入口: notification_type=TYPE-1 直接 bypass LLM 分析
- classify_notification() 優先讀 incident.notification_type (早期分診結果)
- ConfigurationDrift/KubeConfigDrift 補入 TYPE-4D 匹配清單

3-2: infrastructure → SSH MCP routing
- _auto_execute() 中 alert_category=infrastructure + 非 kubectl action → _ssh_execute()
- _ssh_execute(): docker_restart / service_restart tool 路由
- 取 instance label 對應 SSH_MCP_ALLOWED_HOSTS 白名單主機

3-3: send_info_notification() TYPE-1 已存在，classify_notification 修復確保正確呼叫

3-4: Dynamic button builder 已存在 _build_inline_keyboard + _CATEGORY_BUTTONS

3-5: action | parse fix
- _auto_execute() 開頭: action 含 | 時取第一段 (LLM 有時輸出 "kubectl X | kubectl get")

3-6: risk_level YAML priority override LLM
- dual_engine_analyze() LLM 結果返回後，用 alert_rules.yaml 對應 rule.risk 覆蓋

3-7: send_drift_card() TYPE-4D 已存在，classify_notification 修復確保正確觸發

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:39:19 +08:00
OG T
184b37a8b1 refactor(decision_manager): I2 DI 化 MCP Providers + fix config list type bug 
- DecisionManager.__init__ 注入 SSHProvider/K8sProvider，移除函數內 import+實例化
- config.get_tg_user_whitelist() 支援 list 輸入（monkeypatch/直接傳入），修復 AttributeError
- LOGBOOK 更新（test fix 6e0ee8b）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 13:04:46 +08:00
OG T
8be87b0f32 fix(review): 首席架構師 Code Review — c439277 Tier 3 紅區修補
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m39s
Details 
Critical:
- C1: decision_manager _collect_mcp_context container 變數 Python ternary 優先度 bug 修正
  原: `A or B or C[0] if list else ""` (ternary 控制全式)
  修: `A or B or (C[0] if list else "")` (明確括號)
- C2: 所有 MCP 呼叫加 asyncio.wait_for timeout=5s，防止阻塞決策主路徑
  同時加 unknown host warning log (C4)
- C3+M1: _DESTRUCTIVE_PATTERNS 補全移至模組頂層常量
  新增: delete pods(複數)/kubectl drain/kubectl cordon/kubectl rollout undo/
        docker rm/docker stop/docker kill/rm -rf/"replicas": 0(JSON patch)

Important:
- I1: webhooks.py IP 排除改用 is_internal_ip() 支援全 RFC-1918 (10.x/172.16-31.x/192.168.x)
- I4: 新增 test_destructive_patterns.py — 25 測試全過
  涵蓋: 常量存在、攔截、誤攔迴歸、critical 永遠攔截

🔴 Tier 3 紅區 — 首席架構師 Code Review 通過後 push

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 22:05:52 +08:00
OG T
c439277fc3 feat(aiops): ADR-070 全自動化方向 — 三大修復
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
1. auto_approve.py: 允許 high risk 自動執行 (low/medium/high 全開放)
   - min_confidence 0.65→0.50 (信心門檻降低)
   - 新增 DESTRUCTIVE_PATTERNS 攔截真正危險指令
     (scale=0, delete deployment/pvc/namespace, drop table)
   - 核心: critical + 破壞性操作 → 人工; 其他 → 全自動

2. decision_manager.py: 新增 _collect_mcp_context()
   - LLM 分析前先收集真實環境狀態 (SSH/K8s MCP)
   - Host/Docker 告警 → ssh_get_container_status + ssh_get_top_processes
   - K8s 告警 → k8s_get_events
   - 注入 diagnosis_context "當前環境狀態 (MCP 實時查詢)" 區段

3. webhooks.py: 修復 target_resource 提取
   - 新增 name/container/job label 提取
   - DockerContainerUnhealthy 不再 target=alertname
   - IP 位址自動排除 (192.x 開頭不作為 target)

🔴 Tier 3 紅區 — 需首席架構師批准
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:39:52 +08:00
OG T
b2dfcf9b0d fix(telegram): safety guard 攔截改發人工審核卡片，不再發 ❌ 失敗訊息 
問題：AI 無法確認 deployment name 時，每次告警都發一條
「❌ 自動修復失敗 kubectl scale deployment unknown」的垃圾訊息

修復：
- safety guard 攔截 → token.state 回 READY（非 ERROR）
- 改呼叫 _push_decision_to_telegram，發 TYPE-4 人工審核卡片
- mcp_all_failed=True 讓 classify_notification 選 TYPE-4
- K8s 找不到 target 的路徑同樣處理

效果：統帥看到的是「需要人工介入的審核卡片」而非「修復失敗」錯誤訊息

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:33:19 +08:00
OG T
f2c18c4e63 feat(D1): models.json 集中化 — ADR-067 五大 Ollama 應用 hardcode 消除
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m56s
Details 
- models.json v1.3.0: providers.ollama.models 新增 9 個 purpose keys
  (drift_summary/drift_intent/log_anomaly/nemoclaw/playbook_draft/
   code_review/embedding/rag_generate/image_analysis)
- drift_narrator_service: NARRATOR_MODEL → get_model("ollama","drift_summary")
- drift_interpreter: MODEL → get_model("ollama","drift_intent")
- log_summary_service: SUMMARY_MODEL → get_model("ollama","log_anomaly")
- local_code_review_service: _MODEL_OLLAMA → get_model("ollama","code_review")
- image_analysis_service: _MODEL → get_model("ollama","image_analysis")
- decision_manager: nemoclaw + playbook_draft 兩處 → get_model()
- embedding_service: get_embedding_service() factory → get_model("ollama","embedding")
- knowledge_service: OllamaEmbeddingService(model=...) → get_model()

所有模型名稱現在統一由 models.json 管理，修改模型只需改一個檔案。
LOGBOOK 更新：D1 完成 + B2 已完成確認

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:45:53 +08:00
OG T
82e1c05df8 fix(review): Code Review C1/C2/I2/M2 修補
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
C1 drift_interpreter: 寫死 192.168.0.111 → settings.OLLAMA_URL
  違反 feedback_frontend_internal_ip_ban 鐵律（後端 service 層同樣禁止寫死內網 IP）

C2 km_conversion_service: BUG-004 補同步 Redis Working Memory vectorized 欄位
  原修復只更新 DB，Redis incident:{id} JSON 的 vectorized 未同步
  → 審計查 Redis 仍顯示 False，fly-wheel 閉環指標仍不準
  修復：DB 更新後 GET → JSON patch vectorized=True → SET（保留原 TTL）

I2 decision_manager: _ALERTNAME_KEYWORDS HostHighDiskUsage→HostOutOfDiskSpace
  + 補 DockerContainerExited
  + fallback 路徑加 debug log

M2 decision_manager: import json as _json 從 for 迴圈移至方法頂部

docs: ADR-072 新增 Code Review 發現與技術債記錄

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:36:59 +08:00
OG T
e447f97616 fix(telegram): 接通 classify_notification + 修復 HostBackupFailed 亂送按鈕 
三個問題同時修復：

1. classify_notification() 死程式碼接通
   - _push_decision_to_telegram() 現在先呼叫 classify_notification()
   - TYPE-1 (純資訊) → send_info_notification()，無按鈕
   - TYPE-4D (Config Drift) → send_drift_card()
   - 其餘 TYPE-2/3/4 → send_approval_card()（原有按鈕）
   - decision_state + auto_executed 從呼叫端注入 proposal_data

2. alert_rules.yaml 補 host_backup_failed 規則
   - HostBackupFailed / VeleroBackupFailed / VeleroBackupNotRun → NO_ACTION
   - 不再走 generic_fallback → 不再產生 kubectl rollout restart deployment/backup

3. _verify_k8s_deployment_exists() 主機層告警不再保守放行
   - Host*/Docker*/Backup*/Velero*/SSH* 前綴告警 → K8s MCP 不可用時 return False
   - _auto_execute() 收到 NO_ACTION 或空 kubectl_command → 早退，不執行

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:35:48 +08:00
OG T
5aa0244c9a fix(aiops): ADR-072 P1 Bug 修復 — BUG-004/005/006
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
BUG-004 KM vectorization 108/112 = False:
  km_conversion_service: KM entry 建立後（embedding 已背景觸發），
  補寫 incidents.vectorized = True，飛輪閉環（ADR-068）學習指標正常

BUG-005 15 ready decisions 無人審核:
  decision_manager: 新增 resend_stale_ready_tokens()，
  掃描 Redis decision:* key，找出 state=ready 且 dedup_key 過期的 token，
  重新推送 Telegram 審核卡片
  main.py: lifespan startup 排程 resend_stale_ready_tokens()（asyncio.create_task 非阻塞）

BUG-006 outcome/verification_result 全 null:
  _push_auto_repair_result: Telegram 推送前先寫入
  incidents.outcome + incidents.verification_result 到 DB

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:24:41 +08:00
OG T
2185e1755c fix(aiops): ADR-072 P0 Bug 修復 — BUG-001/002/003
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details 
BUG-001 drift_interpreter: nvidia_provider 已重構為 NvidiaProviderResult 物件（非 4-tuple）
  → 改用 Ollama httpx 直接呼叫 qwen2.5:7b-instruct，繞過 nvidia_provider
  → 消除所有 K8s config drift 告警的 "too many values to unpack" 永久失敗

BUG-002 deployment_name="unknown": 主機層告警(HostHighCpuLoad 等)無 component/job/pod label
  → _auto_execute() 新增 _resolve_target_from_k8s() 補救
  → K8s MCP kubectl get pods 動態查詢受影響 Pod，去掉 hash suffix 得到 deployment name

BUG-003 無效 deployment 通過 safety guard:
  → _auto_execute() safety guard 通過後加入 _verify_k8s_deployment_exists() 存在性確認
  → K8s 中找不到 deployment/pod → 拒絕執行，寫入 DecisionToken.error
  → K8s MCP 不可用時保守放行（不阻塞主流程）

2026-04-11 Claude Sonnet 4.6 Asia/Taipei

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:20:39 +08:00
OG T
f3236338a5 fix(security): Code Review P0+P1+P2 全修補 — MCP Phase 2b-3 + decision_manager
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
P0: decision_manager _fetch_metrics_snapshot 參數型別錯誤
  - prom._instant_query(str) → prom._instant_query({"query": str})
  - 結果解析 r.get("status")=="success" → r.get("result", [])

P1: prometheus_provider — alertname PromQL injection 防範
  - 新增 _RE_SAFE_ALERTNAME 白名單正則

P1: decision_manager — kubectl action 危險字元注入防範
  - 新增 _ALLOWED_KUBECTL_PATTERN 白名單，非法指令格式直接拒絕

P1: decision_manager — 6 個 asyncio.create_task() GC 風險
  - 新增 _background_tasks: set + _fire_and_forget() helper
  - 所有 bare create_task 改用 _fire_and_forget

P1: ssh_provider — Group B 寫入工具強制需要 known_hosts
  - known_hosts 未設定或檔案不存在時拒絕執行，防 MITM

P2: sentry_provider — query 語意白名單驗證
  - 新增 _RE_SAFE_SENTRY_QUERY，拒絕含特殊字元的 query

P2: argocd_provider — verify=False 改為 ARGOCD_VERIFY_TLS 環境變數開關
  - 新增 _tls_verify() helper，預設 false（self-signed cert）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:10:33 +08:00
OG T
7eb49f9c20 feat(mcp-phase4c): AI 動態規則生成 — 新 alertname 自動產 Playbook 草稿
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m29s
Details 
_generate_playbook_draft_if_new():
  - Playbook 無命中時非同步觸發（不阻塞決策主流程）
  - 先用 semantic_search(threshold=0.92) 確認 KM 無同名 Playbook
  - 呼叫 qwen2.5:7b-instruct (Ollama 188) 生成五段結構化草稿
    (症狀/根因/診斷步驟/修復動作/驗收條件)
  - 寫入 KnowledgeEntry(type=PLAYBOOK, status=DRAFT, source=AI_EXTRACTED)
  - 寫入 AlertOperationLog PLAYBOOK_DRAFT_CREATED 事件
  - 失敗靜默 debug log

完成 MCP Phase 4 全三項:
  4a NemoClaw second opinion (信心 < 0.7)
  4b K8s 狀態快照 k8s_state_after
  4c AI 動態 Playbook 草稿生成

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:16:39 +08:00
OG T
0fa3b35a1c feat(mcp-phase4b): 自動修復後抓 K8s Pod 狀態寫入 k8s_state_after
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 24s
Details 
_push_auto_repair_result() 成功後:
  - 呼叫 K8sProvider.kubectl_get(pods, label=app=<service>)
  - 結果截斷 500 字寫入 incidents.k8s_state_after
  - km_conversion_service._build_content() 已支援顯示此欄位
  - 失敗靜默 debug log，不阻塞主流程

完成 KM 三段資料閉環: 症狀(labels) + 情境(metrics_before) + 動作(action) + 效果(metrics_after + k8s_state_after)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:15:31 +08:00
OG T
f3ee577f9d feat(mcp-phase4a): NemoClaw second opinion — 信心 < 0.7 觸發 deepseek-r1:14b 複審
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- _nemoclaw_second_opinion(): 呼叫 Ollama 188 deepseek-r1:14b 做獨立推理
  - 解析 <think>...</think> CoT 格式，只取正文
  - 30s timeout，失敗靜默降級
  - 輸出截斷 300 字
- _dual_engine_analyze(): LLM 信心 < 0.7 時非同步觸發 second opinion
  - 結果附加到 proposal_data["advisory_note"]
- _push_decision_to_telegram(): advisory_note 以 NemoClaw bot 身分追加訊息
  - 格式: "NemoClaw 第二意見 (信心=0.xx)"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:14:54 +08:00
OG T
1ec19656b5 feat(adr071-ij): TYPE-2 指標快照卡片 + KM 三段資料整合
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m17s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 36s
Details
Ansible Lint / lint (push) Has been cancelled
Details 
ADR-071-I: decision_manager 執行前後各抓一次 Prometheus metrics
  - _fetch_metrics_snapshot(): 依 alertname 選擇 CPU/Mem/Disk/Restart 查詢
  - _format_metrics_delta(): 輸出 "CPU 92%→23% | Mem 78%→45%" 格式
  - _push_auto_repair_result(): metrics_after 寫 DB + TYPE-2 卡片顯示 delta
  - _auto_execute(): metrics_before 在執行前寫 DB（完成閉環）

ADR-071-J: km_conversion_service._build_content() 使用精簡 delta 格式
  - 從 metrics_before/after 產生人讀 delta（CPU/Mem/Disk/重啟次數）
  - 附加 k8s_state_after（若有）
  - 格式: 症狀 + 根因 + 動作 + 效果數字（症狀→情境→動作→效果）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 03:09:35 +08:00
OG T
68a3858ae4 fix(auto_execute): 守衛加入 target==alertname 檢查，防止 LLM 把告警名稱當 deployment 名稱
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 13m33s
Details 
HostHighCpuLoad 等主機告警，NemoTron Tool Calling 可能把
alertname 填入 deployment_name，導致執行
'kubectl rollout restart deployment HostHighCpuLoad'。

新增守衛: _target == _alertname 時拒絕執行並通知人工介入。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 01:13:24 +08:00
OG T
a4d655ea7f fix(auto_execute): 安全守衛 — 拒絕執行含 unknown 或未解析 placeholder 的 action
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 19m7s
Details
E2E Health Check / e2e-health (push) Successful in 43s
Details 
主機層告警（HostHighCpuLoad、DockerContainerUnhealthy 等）沒有對應
K8s deployment 名稱，affected_services=[]，導致 _target='unknown'，
執行 'kubectl rollout restart deployment unknown' 這種無意義命令。

修復: 替換後若 action 仍含 'unknown' 或 <...>/{...} 格式，
直接拒絕執行並通知人工介入，不允許帶 placeholder 的命令上線。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 23:57:17 +08:00
OG T
f8926bb70a ci: 觸發 CD — decision_manager 修復標記 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 22:12:56 +08:00
OG T
e5f1541d69 fix(auto_execute): 替換 action 中的 <deployment_name>/{target}/{namespace} placeholder
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 24s
Details 
Nemotron tool calling 生成 <deployment_name> 佔位符未替換
auto_execute 前統一替換所有 {target}/{namespace}/<xxx> 格式

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 22:00:19 +08:00
OG T
71f0dbf2b5 fix(auto_execute): ApprovalRequest 補齊 description/requested_by/required_signatures
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
3 validation errors 導致 auto_execute_failed
補上所有必填欄位，required_signatures=0 表示自動核准不需簽核

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 21:59:52 +08:00
OG T
167e115a6d feat(phase31): Log 異常摘要觸發點 — 告警後 NemoTron 發 log summary
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m44s
Details 
_send_log_summary: 取 Pod log → deepseek-r1:14b 分析 → NemoTron 發到群組
觸發點: _push_decision_to_telegram 送完審批卡後異步執行

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 16:07:56 +08:00
OG T
1483218bab feat(approval): 批准/拒絕後立即回應 Telegram + 持久化 message_id 到 DB
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 13m9s
Details 
問題：按下 TG 批准/拒絕按鈕後完全沒有任何回應，使用者不知道是否成功。
      Telegram message_id 只存 Redis 24h TTL，過期後無法追蹤。

修正：
- approval_records 加 telegram_message_id / telegram_chat_id 欄位（已 ALTER TABLE）
- approval_db.update_telegram_message() — 持久化 message_id 到 DB
- decision_manager: 發送告警卡片後同時寫 Redis + DB
- telegram_gateway._notify_approval_result() — 批准/拒絕後：
    1. editMessageReplyMarkup 移除批准/拒絕按鈕，保留資訊按鈕
    2. sendMessage reply_to 在原訊息下回覆狀態行
    3. fallback: send_notification 發新訊息
- _handle_group_command: chat_id 改為 _chat_id 消除 IDE lint

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 18:19:31 +08:00
OG T
d8c2969341 feat(telegram): AI 鏈路透明化 — 告警訊息顯示 OpenClaw + Tool Calling 模型/後端
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m12s
Details 
- nemotron.py: 偵測 OllamaToolProvider vs NvidiaProvider，記錄 tool_model/tool_backend
- openclaw.py: 傳播 nemotron_tool_model/nemotron_tool_backend 到 proposal
- decision_manager.py: 從 proposal_data 提取並傳給 send_approval_card()
- telegram_gateway.py: TelegramMessage 新增兩個欄位，format_with_nemotron 顯示
  "🔧 Tool Calling: llama3.1:8b (Ollama 本機)" 或 "NVIDIA 雲端"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 15:05:16 +08:00
OG T
4f80ba38c0 feat: 告警狀態變更在原訊息延續 (方案 B)
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m28s
Details 
**telegram_gateway.py**
- 新增 append_incident_update(incident_id, status_line)
  - 從 Redis tg_msg:{id} 取 message_id
  - editMessageReplyMarkup: 移除 Row1(批准/拒絕/靜默)，保留 Row2(詳情/重診/歷史)
  - sendMessage reply_to_message_id: 在原訊息下方追加狀態行
  - 找不到 message_id 回傳 False（呼叫方自行 fallback）

**decision_manager.py**
- _push_decision_to_telegram: send_approval_card 後存 tg_msg:{id}=message_id (TTL 24h)
- _push_auto_repair_result: 改用 append_incident_update，找不到 message_id 才 fallback 新訊息

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 14:21:33 +08:00
OG T
2554ac1e60 fix: E2E test 告警識別 + 自動修復結果 Telegram 通知
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
**alert_rule_engine.py**
- _matches() 加入 instance_prefix 匹配（最高優先）
- match_rule() 傳入 instance label 至 _matches
- 用途: e2e-final-* / e2e-test-* instance 可被 YAML 規則識別

**alert_rules.yaml**
- 新增 e2e_smoke_test 規則 (priority=120)
- alertname: E2E_SMOKE_TEST / instance_prefix: e2e-final- / e2e-test- / test-host
- suggested_action: NO_ACTION，顯示「告警鏈路驗證成功」

**decision_manager.py**
- _auto_execute() 成功後發 Telegram 結果通知 ✅
- _auto_execute() 失敗後發 Telegram 失敗通知 ❌
- 新增 _push_auto_repair_result() 函數

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 14:16:15 +08:00
OG T
b66263ad36 fix(decision_manager): resolved Incident 不重送 Telegram
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details 
dedup TTL 10分鐘過期後，已 resolve 的 Incident 仍被重新推送
加入狀態檢查，resolved/closed 直接跳過

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 12:00:39 +08:00
OG T
9361fd1fa7 fix(decision_manager): action 不應 strip_placeholders 避免截斷 deployment name
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
_strip_placeholders 移除 <...> 導致 kubectl rollout restart deployment/<name>
變成 kubectl rollout restart deployment/，Telegram 顯示建議指令不完整

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 11:45:33 +08:00
OG T
c5e475121a fix(telegram): 修復建議指令被截斷 + decision_manager enum string 補正
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
根因 1: telegram_gateway.py suggested_action[:35] 剛好截到 deployment/ 後
  → 改為 [:80]，完整顯示 kubectl command

根因 2: 舊 Incident proposal_data 存 enum string (RESTART_DEPLOYMENT)
  → decision_manager.py 加入偵測，用規則引擎重新查 kubectl command

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 11:14:30 +08:00
OG T
4935cfc346 fix(telegram): 重設計訊息格式 + 修復 detail/reanalyze/history 按鈕失效
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m26s
Details 
- format() / format_with_nemotron(): 移除 ═══ 分隔符，改為簡潔換行佈局
- send_approval_card(): 新增 incident_id 參數，傳入 _build_inline_keyboard()
- decision_manager.py: 呼叫 send_approval_card() 時傳入 incident.incident_id
- 問題根因: incident_id 未傳入 _build_inline_keyboard() 導致第二排按鈕從未渲染

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 12:44:13 +08:00
OG T
429d81d29b fix(knowledge): I2+I3 首席架構師 Important 修復 — 依賴注入 + exception 細分
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
I2: KnowledgeService 移至 DecisionManager.__init__ 注入
    _query_kb_context_inner 使用 self._knowledge_svc，移除函數內 import 耦合

I3: _query_kb_context exception 細分
    - asyncio.TimeoutError → warning (預期降級)
    - ConnectionError/OSError → warning (Ollama 連線問題，預期降級)
    - Exception → error (非預期，提升監控可見性)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-04 11:51:43 +08:00
OG T
f846000c8c fix(knowledge): C1 首席架構師必修 — _query_kb_context 5秒 hard timeout
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
C1 修復 (首席架構師 Review 74/100 → 條件通過):
- 抽出 _query_kb_context_inner 含實際查詢邏輯
- _query_kb_context 用 asyncio.wait_for(timeout=5.0) 包裝
- Ollama hang/慢響應最多消耗 5s，保護 30s 決策 SLA
- timeout 時 logger.warning("kb_rag_timeout") 靜默降級

同步移除 LLM prompt 中的 emoji (## 📚 → ## Knowledge Base)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-04 11:48:57 +08:00
OG T
860dc1d892 feat(knowledge): KB Phase 2 — OpenClaw RAG 整合
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
_dual_engine_analyze 新增 _query_kb_context():
- Incident 分析前語意搜尋相關 KB 條目 (top-3, threshold=0.4)
- 將 KB context 注入 expert_context.diagnosis_context 傳給 LLM
- 失敗時靜默降級，不影響主分析流程
- dual_engine_llm_win log 新增 kb_rag 欄位，可觀測 RAG 命中率

架構: _query_kb_context 透過 get_knowledge_service() 呼叫 Service 層
符合 leWOOOgo 積木化 — decision_manager 不直接存取 DB/pgvector

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-04 11:46:47 +08:00
OG T
d0f09705e5 fix(auto-repair): 修復三個阻礙自動修復的根本原因
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
1. playbook_rag: Ollama embedding http_client 滾動重啟後 is_closed
   - 新增 _get_http_client() 偵測 is_closed 自動重建
   - singleton get_playbook_rag_service() 加 is_closed 重建判斷

2. telegram: 加入 ai_model 欄位顯示底層判斷模型
   - TelegramMessage.ai_model 欄位
   - format() / format_with_nemotron() 顯示 "Nemotron (nemotron-70b)"
   - openclaw proposal_dict 加入 model 欄位
   - decision_manager / send_approval_card 串接

3. DB: 清除 9 筆 3/26 殭屍 PENDING (mock_fallback CRITICAL 測試記錄)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-04 11:46:25 +08:00
OG T
d2f02999b7 fix(alert-format): 移除 [LLM_OPENCLAW_NEMO] prefix + 擴大根因/建議字數
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 7m4s
Details 
- root_cause: 移除 [source.upper()] 前綴，直接顯示 AI 分析文字
- root_cause 截斷: 80→150 字
- suggested_action 截斷: 50→80 字
- AI provider 來源已在訊息標頭 「🤖 OpenClaw Nemo 仲裁」顯示，不需在根因重複

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-03 17:43:19 +08:00
OG T
d32d84efce feat(telegram): 接通 Phase 22 Nemotron 雙軌顯示 (ADR-044)
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
E2E Health Check / e2e-health (push) Has been cancelled
Details 
根本原因: format_with_nemotron() 已實作但從未被呼叫
- send_approval_card() 新增 nemotron_enabled/tools/validation/latency 參數
- TelegramMessage 建構時傳入 nemotron 欄位
- nemotron_enabled=true 時自動使用 format_with_nemotron() 格式
- _push_decision_to_telegram() 從 proposal_data 提取並傳遞 nemotron 資料

效果: Telegram 同時顯示 OpenClaw 仲裁 + Nemotron 執行方案雙區塊
2026-04-02 ogt: Phase 22 最後一哩路

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 08:59:03 +08:00
OG T
27b4d2a76a fix(telegram): strip <placeholder> 佔位符防止 HTML parse 錯誤
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
E2E Health Check / e2e-health (push) Has been cancelled
Details 
OpenClaw 生成的 kubectl_command 含 <受影響服務名稱>
在 Telegram HTML parse mode 下造成 'Can't parse entities'
用 regex strip 所有 <...> 佔位符

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 22:50:07 +08:00
OG T
dd526684ab feat(ai): Phase 22 OpenClaw + Nemotron 協作架構 (ADR-044)
All checks were successful
E2E Health Check / e2e-health (push) Successful in 17s
Details 
統帥批准實作「仲裁-執行分工」架構:
- OpenClaw = 仲裁者 (Why + Risk Level)
- Nemotron = 執行者 (How + kubectl Command)

新增功能:
- config.py: ENABLE_NEMOTRON_COLLABORATION Feature Flag
- openclaw.py: generate_incident_proposal_with_tools()
- openclaw.py: _call_nemotron_tools() Nemotron 呼叫
- telegram_gateway.py: TelegramMessage Nemotron 欄位
- telegram_gateway.py: format_with_nemotron() 雙區塊格式
- decision_manager.py: 整合協作方法
- proposal_service.py: 整合協作方法

觸發條件:
- LOW 風險 → 僅 OpenClaw
- MEDIUM/HIGH/CRITICAL → OpenClaw + Nemotron 雙軌

首席架構師審查: 83/100 條件通過

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-31 18:52:53 +08:00
OG T
b97f9364fb feat(k8s): add Worker HPA + fix non-AI confidence values 
Wave 2 Deployment:
- Worker HPA: min:1 max:3, CPU 70%, Memory 80%
- 前置條件: XCLAIM + terminationGracePeriodSeconds:90 (Wave 1 ✅)
- 比 API/Web 更保守的擴縮策略 (120s up, 600s down)

Confidence Fix:
- 非 AI 分析來源 (fallback/playbook/historical/consensus) 設 confidence=0.0
- 避免混淆 AI 信心度與其他指標 (成功率/相似度)
- 涉及: github_webhook, decision_manager, intent_classifier, learning_service

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 16:09:37 +08:00
OG T
5f9a6a7e55 fix(ai): 移除假信心分數 + 顯示 AI 模型來源 
問題: AI 仲裁顯示硬編碼信心分數 (0.75/0.88/0.92/0.70)

修復:
- decision_manager: 預設 confidence 0.75 → 0.0
- decision_manager: Expert System confidence=0.0 + is_rule_based
- openclaw: 所有 Mock Response confidence → 0.0
- telegram_gateway: 新增 ai_provider 欄位
- telegram_gateway: 動態來源標籤 (Ollama/Gemini/Claude/規則匹配)

Telegram 卡片顯示:
- confidence > 0 + provider=ollama → 🤖 Ollama 仲裁
- confidence > 0 + provider=gemini → 🤖 Gemini 仲裁
- confidence > 0 + provider=claude → 🤖 Claude 仲裁
- confidence == 0 → ⚙️ 規則匹配

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:19:51 +08:00
OG T
138ef0c2db fix(api): 修復 7 個 Lint 錯誤 (unused imports + zip strict + dict comprehension) 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-27 14:42:47 +08:00
OG T
abc21c735e feat(api): P1 Telegram 按鈕優化 - 稍後/靜默 
新增按鈕:
- ⏰ 稍後 (snooze): 延遲 30 分鐘後再提醒
- 🔕 靜默 1h (silence): 同類資源告警靜默 1 小時

實作細節:
- telegram_gateway.py: 新增 _handle_snooze/_handle_silence
- decision_manager.py: 發送前檢查 silence 狀態
- Redis Key: telegram_snooze:{approval_id}, telegram_silence:{resource_name}
- Skill 03 v1.5 → v1.6

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-27 09:50:28 +08:00
OG T
e34b0f2e9a fix(api): Telegram 去重 + 修復 INC-INC-INC- 重複前綴 
- 加入 Redis 去重機制 (10 分鐘 TTL)
- 修復 approval_id 重複添加 INC- 前綴

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-27 09:27:40 +08:00
OG T
855b7f5593 fix: 修復 Telegram 轟炸問題 
問題: 765ee39 的修改導致 COMPLETED 狀態下 incident 未解決時
會建立新 decision，每次 poll 都觸發 Telegram 發送

修復: COMPLETED 狀態直接返回 existing_token，不建立新 decision

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-26 22:24:03 +08:00
OG T
ce7f8a1b23 feat(api): ADR-030 Phase 4 自動執行機制 
實作低風險操作自動執行策略：

1. auto_approve.py - 自動執行策略服務
   - AutoApprovePolicy: 評估是否可自動執行
   - 條件: LOW 風險 + 信任分數 >= 5 + Playbook 成功率 >= 95%
   - CRITICAL 永遠不自動執行
   - 完整審計追蹤

2. trust_engine.py - 新增 singleton
   - get_trust_manager(): 取得全域 TrustScoreManager

3. decision_manager.py - 整合自動執行 (Tier 3 紅區)
   - Step 5 加入 AutoApprovePolicy 判斷
   - 條件滿足時跳過 Telegram，直接執行
   - _auto_execute(): 自動執行邏輯
   - 失敗時 fallback 到人工審核

流程:
Incident → 分析 → AutoApprovePolicy 評估
  ├─ 可自動執行 → 直接執行 → 完成
  └─ 需人工審核 → Telegram 通知 → 等待批准

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-26 22:13:10 +08:00
OG T
17ee8838be revert: 還原 Telegram + CD 到正常狀態 
還原檔案到 d071019 版本：
- decision_manager.py: 移除 Redis dedup 邏輯
- telegram_gateway.py: 還原 INC- 前綴邏輯
- cd.yaml: 移除 selector immutable 處理和 Token injection

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-26 22:10:52 +08:00
OG T
60e9538889 feat(api): ADR-030 Phase 2 診斷資料收集強化 
實作智能自動修復系統的資料收集層：

1. k8s_diagnostics.py - K8s 診斷服務
   - Pod Events/Logs/ResourceUsage 收集
   - CrashLoopBackOff/OOM/ImagePull 偵測
   - 非同步並行收集 + 錯誤容忍

2. diagnosis_aggregator.py - 診斷聚合器
   - 整合 K8s + SignOz + Expert Rules
   - DiagnosisContext 提供結構化 LLM Prompt
   - DiagnosisSignal 信號分析

3. decision_manager.py - 決策引擎整合
   - Step 2.5 加入診斷收集
   - 傳遞 diagnosis_context 給 LLM

4. openclaw.py - LLM Prompt 增強
   - 整合 K8s/SignOz 深度診斷上下文
   - 支援 diagnosis_signals 摘要

ADR-030 架構: 診斷先行，根因分析，非盲目重啟

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-26 21:55:50 +08:00
OG T
bb6151cf44 revert: 移除 Telegram Redis dedup 邏輯 
原因: dedup 邏輯導致 Telegram 完全無法發送
保留: INC- 前綴修復 (approval_id = incident.incident_id)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-26 21:53:39 +08:00
OG T
d148756b67 feat(api): LLM 整合 Expert System 診斷上下文 
長期方案實作: Expert 診斷 + LLM 智能分析

變更:
1. decision_manager._dual_engine_analyze():
   - 測試資源跳過 LLM (省錢)
   - 傳遞 Expert 診斷上下文給 LLM
   - LLM 失敗時根據診斷調整回應

2. openclaw.generate_incident_proposal():
   - 新增 expert_context 參數
   - Prompt 包含 Expert 診斷結果
   - 引導 LLM 基於診斷做決策

流程:
Playbook → Expert診斷 → LLM(with context) → 智能建議

這是「先診斷根因，再決定行動」的正確實作

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-26 21:41:26 +08:00