wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity
1,398 Commits 203 Branches 2 Tags
fab65e7d7a5e550e07cf5748b27fedb01e369069
Commit Graph

12 Commits

Author SHA1 Message Date
OG T
6c7f648b60 fix: 3 個飛輪沉默未打通節點 — 統帥截圖盤出
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 18m56s
Details 
統帥截圖證據 (Telegram MEDIUM 告警仍走人工審核):
INC-20260411-A03B2E / A2BB29 顯示「[規則匹配]」+ action=unknown-service

節點 1: AutoApprovePolicy 擋下規則匹配 (飛輪主因)
  - ADR-073 規則匹配 confidence=0.0 (防偽造)
  - AutoApprovePolicy.min_confidence=0.50 → 擋下
  - 結果: MEDIUM 規則匹配永遠人工審核，飛輪不轉
  修復: auto_approve.py 加 _is_rule_based 判斷
        (is_rule_based / source=expert_system / rule_id / matched_rule)
        → bypass min_confidence 檢查
        → 驗證: should_auto_approve=True ✅

節點 2: _is_bad_target 漏 unknown-service magic string
  - _resolve_target_from_k8s fallback 產 unknown-service / unknown-pod
  - GAP-A4 Phase 1/2 只擋 'unknown' 而非前綴
  修復: alert_rule_engine.py 加 unknown-/none-/null-/undefined- 前綴黑名單
        → 驗證: 4 個 magic 全 bad ✅

節點 3: stale_ready_tokens_resend 無時效過濾
  - 截圖是 2026-04-11 (4 天前) 告警
  - 舊 labels 過期，重 process 也產不出新 target
  - 壓爆 Ollama + 污染 Telegram 卡片
  修復: decision_manager.py 跳過 > 3 天的 stale incident
        → skip + log stale_ready_token_skipped_too_old

回歸: 113/113

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2026-04-15 10:56:48 +08:00
OG T
10b74affcf fix(GAP-A4): 規則 Action 模板 placeholder 解析修復 — 解開 8.3h 飛輪沉默
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
🚨 真因診斷（統帥逮到）:
API log 顯示最近 1 小時爆發大量 auto_execute_blocked_unresolved_placeholder:
  - action: "kubectl rollout restart deployment HostHighCpuLoad"  ← target=alertname
  - action: "kubectl rollout restart deployment unknown"
  - action: "kubectl scale deployment unknown --replicas=3"

根因：alert_rule_engine._extract_vars() target 解析邏輯不夠強健，
當 Prometheus 告警無 deployment label 時，退回 alertname 或 "unknown"，
產生垃圾指令。GAP-A1 防注入閘正確攔下，但自動修復路徑因此卡死，
KM 不寫入 → 飛輪沈默。

修復（三層防護）:

1. 新增 _strip_pod_suffix() — K8s Pod 名稱還原 Deployment base
   - Deployment 格式: awoooi-api-7d6b776f78-4sgjl → awoooi-api
   - StatefulSet: postgresql-0 → postgresql
   - Legacy: my-job-x2m4k → my-job

2. 新增 _is_bad_target() — 垃圾 target 識別
   - 空串 / "unknown" / "none" / "null"
   - target == alertname 本身
   - IP:port 格式、純 IP、含空白/括號/引號
   - 未解析 {placeholder}

3. 重寫 _extract_vars() — 多層 label 查找（權威優先）:
   deployment > app > statefulset > pod(去後綴) > container > service > target_resource
   每層都過 _is_bad_target 驗證，全失敗 → target="unknown"

4. match_rule() 後置雙驗證:
   - bad target → 清空 kubectl_command (降級 LLM)
   - 殘留 { or } → 清空 kubectl_command (模板未填完)

測試覆蓋：
- 33 個新單元測試（GAP-A4 四大場景全覆蓋）
- 214/214 回歸測試全過

影響：
- 原本產出「kubectl rollout restart deployment HostHighCpuLoad」的路徑
  → 現在會 `rule_kubectl_command_discarded_bad_target` 並降級 LLM
- LLM 若能從錯誤 log 推理真實 deployment，飛輪恢復正常運轉
- 若 LLM 也無解，進 TYPE-4 人工扶梯

2026-04-14 Claude Sonnet 4.6（MASTER 藍圖之外的隱性 Bug 殲滅）

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
 2026-04-14 18:43:29 +08:00
OG T
cc42aa0bdb feat(adr-076): Task 2.2 + 2.3 — 規則擴充 + kubectl 注入防護
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
Task 2.2: alert_rules.yaml 新增 3 類規則 (priority 125-127)
  - gitea_down: Gitea CI/CD 下線 → NO_ACTION (priority 125, critical)
  - ssl_cert_expiring: SSL 憑證到期 → NO_ACTION (priority 126, medium)
  - external_site_down: MoWoooWork/Dev/Blackbox probe → NO_ACTION (priority 127, medium)
  規則總數: 21 → 24

Task 2.3: alert_rule_engine.py kubectl 注入防護
  - _RULE_ENGINE_DESTRUCTIVE_RE: 阻擋 delete pvc/namespace/statefulset/deployment,
    drain/cordon, --replicas=0, rm -rf, DROP TABLE, $() 反引號
  - validate_kubectl_command(): 公開 API，SSH 指令/空字串直接通過
  - match_rule() 整合: 變數替換後驗證，阻擋時清空 + log warning
  - test_alert_rule_engine_validation.py: 34 tests (100% 通過)

測試: 776 passed, 26 skipped, 0 failed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-14 15:10:10 +08:00
OG T
c09521a1c6 fix(cr): Code Review P0/P1 全修補 — 積木化+SSH路由+安全守衛順序
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m30s
Details 
P0-1: classify_alert_early 移至 incident_service (Service層)，webhooks.py import 修正
P0-2: _ssh_execute() 改用 self._ssh，移除冗餘 SSHProvider() 實例化
P1-1: infrastructure SSH routing 移至 kubectl safety guard 之前，docker指令不再被攔截
P1-2: alert_rule_engine 新增 get_risk_for_alertname() public API
P1-3: classify_notification() docstring 修正 ORM→Pydantic

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:51:12 +08:00
OG T
d77b2add73 fix(review): 首席架構師 Code Review 修補 — I1 get_incident_type 邏輯修正 + 測試補全
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m13s
Details 
Code Review 發現 2 個 Critical + 2 個 Important 問題：

Critical：
- rule.id 語意為「規則識別符」，與 incident_type 命名空間不同，不可混用
  移除 rule_id fallback 路徑，YAML 匹配無 incident_type 時 fall through 靜態 dict
- get_incident_type() 關鍵路徑無測試覆蓋
  新增 test_get_incident_type.py：11 測試、4 類別（靜態/YAML優先/YAML錯誤/custom）全過

Important：
- ALERTNAME_TO_TYPE deferred import 移至模組頂層（無 circular 風險）
- alert_types.py TODO 過期 → 更新為 I1 整合後正確說明

技術債記錄：NetworkPolicy ArgoCD egress ClusterIP 10.43.16.201/32 需 ArgoCD 重裝後更新

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:33:19 +08:00
OG T
615822dcf3 feat(I1): ADR-064 Rule Engine 整合 — 動態推斷 incident_type
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 11m28s
Details 
- alert_rule_engine.py: 新增 get_incident_type(alertname)
  優先從 YAML 規則 match.alertname 查找 incident_type/rule_id
  Fallback: ALERTNAME_TO_TYPE 靜態 dict → "custom"
- webhooks.py: alert_type 改用 get_incident_type(alertname)
  取代 ALERTNAME_TO_TYPE.get() 靜態查找
- YAML 規則 19 條 alertname 覆蓋自動生效（無需手改 dict）
- 新 alertname 觸發 generic_fallback → auto_generate_rule() 後自動加入 YAML

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:21:41 +08:00
OG T
2554ac1e60 fix: E2E test 告警識別 + 自動修復結果 Telegram 通知
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
**alert_rule_engine.py**
- _matches() 加入 instance_prefix 匹配（最高優先）
- match_rule() 傳入 instance label 至 _matches
- 用途: e2e-final-* / e2e-test-* instance 可被 YAML 規則識別

**alert_rules.yaml**
- 新增 e2e_smoke_test 規則 (priority=120)
- alertname: E2E_SMOKE_TEST / instance_prefix: e2e-final- / e2e-test- / test-host
- suggested_action: NO_ACTION，顯示「告警鏈路驗證成功」

**decision_manager.py**
- _auto_execute() 成功後發 Telegram 結果通知 ✅
- _auto_execute() 失敗後發 Telegram 失敗通知 ❌
- 新增 _push_auto_repair_result() 函數

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 14:16:15 +08:00
OG T
79a9a514dd fix(rules): ADR-064 L1 Redis 分散式鎖防止多 Pod 重複生成規則
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details 
問題: _generating set 是進程級，多 Pod 各自獨立，同一 alertname 可能被
  多個 Pod 同時送給 Ollama/Gemini 生成規則

修復: SET NX EX lock_key — 只有第一個 Pod 能取鎖，其他 Pod 直接跳過
降級: Redis 不可用時 fallback 進程級 set（保持原有行為）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 12:03:51 +08:00
OG T
428e66c111 fix(arch-review): 首席架構師審查 S1×3 S2×3 S3×3 全修復 + ADR-064
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
S1 Critical:
- S1-1: asyncio 觸發移至 _call_with_fallback async 上下文，移除 sync 中的 get_event_loop()
- S1-2: _append_rule_to_yaml 加 textwrap.dedent() 正規化 LLM 輸出縮排
- S1-3: _matches() 對 alertname=["*"] 直接回傳 False，防意外命中

S2 Major:
- S2-1: auto_generate_rule() 改為 DI 參數注入 (ollama_url/model/gemini_api_key)，移除 import settings
- S2-4: _generate_mock_response docstring 澄清為規則引擎生產路徑，非假數據
- S2-5: suggested_action .strip() 防空白字串繞過 or

S3 Minor:
- S3-2: priority 上界 min(next, 890)
- S3-3: alertname sanitize re.sub([{}]) 防 format KeyError
- S3-4: model_registry.py 最後修改時間戳更新

文件:
- ADR-064: Alert Rule Engine YAML 驅動 + AI 自動學習
- Skills 02: 告警規則引擎 DI 規範 + asyncio 禁止事項
- Skills 03: _generate_mock_response 語意澄清 + 規則引擎降級流程
- LOGBOOK: 本次 Session 完整記錄

2026-04-09 ogt: 首席架構師審查修正

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 10:52:40 +08:00
OG T
89da2d24be fix(model-registry): fallback config 更新為 deepseek-r1:14b + gemma3:4b
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 13m20s
Details 
- model_registry._get_default_config: ollama summary llama3.2:3b → gemma3:4b
- model_registry._get_default_config: ollama default/rca → deepseek-r1:14b
- 修正 test_smart_router::test_simple_context 失敗 (斷言 gemma3:4b)
- alert_rule_engine: 移除 asyncio/time unused import
- 2026-04-09 ogt

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 09:52:47 +08:00
OG T
71437db0e9 feat(rule-engine): 自動規則生成 — generic_fallback 觸發 AI 學習
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 11m25s
Details 
流程:
1. 告警命中 generic_fallback 規則
2. 背景觸發 auto_generate_rule()
3. Ollama (deepseek-r1:14b) 生成 YAML 規則片段
4. Ollama 失敗 → Gemini 備援
5. 驗證格式 → append alert_rules.yaml → 清除 lru_cache
6. 下次同類告警直接命中專屬規則，不再走兜底

去重: 同一 alertname 進程內只生成一次
手寫規則 priority 1-499，AI 生成 500-899，兜底 999

2026-04-09 ogt: AI 自學規則引擎

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 09:20:33 +08:00
OG T
d1ede7f989 feat(openclaw): 告警規則引擎 — alert_rules.yaml 取代硬編碼 if/elif
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- 新增 alert_rules.yaml: 6 條規則 (docker/target_down/oom/cpu/5xx/crash) + 通用兜底
- 新增 alert_rule_engine.py: YAML 載入、匹配邏輯、變數填充
- openclaw.py _generate_mock_response: 重構為呼叫規則引擎 (v8.0)
- 新增規則只需修改 YAML，重啟 Pod 即可，不需改代碼
- 2026-04-09 ogt: 架構重構

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 09:05:23 +08:00