feat(ci): Phase 12.3 Ollama 自動化測試 (#67-68)

新增: - CI Ollama Model Test job (連線測試 + 冒煙測試) - test_model_regression.py (4 個回歸案例 + 準確度報告) - Skills 03 更新模型選擇規則 Phase 12.1-12.2 完成記錄更新 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-03-25 11:26:10 +08:00
parent c437b0c749
commit 0a1787e934
4 changed files with 341 additions and 5 deletions
--- a/.agents/skills/03-openclaw-cognitive-expert.md
+++ b/.agents/skills/03-openclaw-cognitive-expert.md
@@ -235,6 +235,59 @@ grep -A 5 "def parse_operation_from_action" apps/api/src/api/v1/approvals.py

 ---

+## 🤖 本地模型選擇策略 (Phase 12.2)
+
+> **更新**: 2026-03-25 | **硬體**: 192.168.0.188 (62GB RAM, 無 GPU)
+
+### Tier 分層選擇
+
+```
+告警進入
+    ↓
+[判斷複雜度]
+    │
+    ├─ 簡單 (單一指標) → llama3.2:3b (~2GB, 快速)
+    │
+    ├─ 中等 (多指標) → qwen2.5:7b-instruct (~5GB, 平衡)
+    │
+    └─ 複雜 (跨服務) → qwen2.5:14b-instruct-q4 (~10GB, 深度)
+```
+
+### 模型配置
+
+| 場景 | 推薦模型 | 參數 |
+|------|----------|------|
+| 中文告警分析 | qwen2.5:7b-instruct | 中文優化 |
+| 快速摘要 | llama3.2:3b | 低延遲 |
+| 複雜提案 | qwen2.5:14b-instruct-q4 | 高準確度 |
+| 推理鏈分析 | deepseek-r1:7b-qwen-distill | 備選 |
+
+### 調用規範
+
+```python
+# ✅ 正確 (根據複雜度選模型)
+async def get_model_for_task(complexity: str) -> str:
+    model_map = {
+        "simple": "llama3.2:3b",
+        "medium": "qwen2.5:7b-instruct",
+        "complex": "qwen2.5:14b-instruct-q4_K_M",
+    }
+    return model_map.get(complexity, "qwen2.5:7b-instruct")
+
+# ❌ 禁止 (硬編碼單一模型)
+model = "qwen2.5:72b"  # 超過硬體限制
+```
+
+### 硬體限制
+
+| 資源 | 約束 |
+|------|------|
+| 最大模型 | 32B (Q4 量化) |
+| 推薦模型 | 7B-14B |
+| ☁️ Cloud 模型 | 移至 Phase 12.4 評估 |
+
+---
+
 ## 參考文檔

 - `apps/api/src/services/incident_engine.py`: 聚合引擎
--- a/.github/workflows/ci.yaml
+++ b/.github/workflows/ci.yaml
@@ -215,6 +215,54 @@ jobs:
          uv run pytest tests/ --cov=src --cov-report=xml -v || true
        continue-on-error: true

+  # ==================== Ollama Model Test (Phase 12.3 #67) ====================
+  # 🤖 自動化模型回歸測試 - 確保 OpenClaw 提案品質
+  ollama-test:
+    name: Ollama Model Test
+    runs-on: [self-hosted, harbor, k8s]
+    needs: api-lint
+    timeout-minutes: 5
+    continue-on-error: true  # 不阻塞主 Pipeline
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Test Ollama Connectivity
+        run: |
+          echo "🔗 測試 Ollama 連線..."
+          if curl -s --connect-timeout 5 http://192.168.0.188:11434/api/tags > /dev/null; then
+            echo "✅ Ollama 可達"
+            curl -s http://192.168.0.188:11434/api/tags | jq -r '.models[].name'
+          else
+            echo "⚠️ Ollama 無法連線 (192.168.0.188:11434)"
+            exit 0  # 不失敗，只警告
+          fi
+
+      - name: Model Smoke Test
+        run: |
+          echo "🧪 模型冒煙測試..."
+          RESPONSE=$(curl -s --max-time 60 http://192.168.0.188:11434/api/generate -d '{
+            "model": "qwen2.5:7b-instruct",
+            "prompt": "你是 AIOps 助手。回答：1+1=?",
+            "stream": false
+          }' | jq -r '.response // "ERROR"')
+
+          if [ "$RESPONSE" != "ERROR" ] && [ -n "$RESPONSE" ]; then
+            echo "✅ 模型回應正常"
+            echo "回應: $RESPONSE"
+          else
+            echo "⚠️ 模型回應異常"
+          fi
+
+      - name: Action Parsing Test
+        working-directory: apps/api
+        env:
+          PYTHONPATH: ${{ github.workspace }}/apps/api
+        run: |
+          echo "🔍 Action Parsing 回歸測試..."
+          # 只執行 action parsing 測試
+          uv sync
+          uv run pytest tests/test_action_parsing.py -v --tb=short || echo "⚠️ 部分測試失敗"
+
  # ==================== OpenAPI Validation ====================
  openapi-validate:
    name: OpenAPI Validate
--- a/apps/api/tests/test_model_regression.py
+++ b/apps/api/tests/test_model_regression.py
@@ -0,0 +1,194 @@
+"""
+Phase 12.3: Model Regression Tests (#68)
+=========================================
+自動化模型回歸測試，確保 OpenClaw 提案品質
+
+測試維度:
+1. 回應語言 (繁體中文)
+2. 命令格式 (kubectl)
+3. 風險評估準確度
+"""
+
+import asyncio
+import os
+import httpx
+import pytest
+from typing import Any
+
+# Ollama 伺服器配置
+OLLAMA_URL = os.getenv("OLLAMA_URL", "http://192.168.0.188:11434")
+DEFAULT_MODEL = os.getenv("OLLAMA_MODEL", "qwen2.5:7b-instruct")
+TIMEOUT = 120  # 秒
+
+
+# =============================================================================
+# 測試案例定義
+# =============================================================================
+
+REGRESSION_CASES = [
+    {
+        "name": "中文告警分析",
+        "prompt": "你是 AIOps 助手。分析告警：服務 awoooi-api CPU 95%。建議修復行動，用繁體中文，50字內。",
+        "validators": [
+            lambda r: any(c in r for c in ["建議", "執行", "擴", "重啟", "優化"]),  # 包含行動建議
+            lambda r: len(r) < 200,  # 回應簡潔
+        ],
+        "description": "應包含中文行動建議",
+    },
+    {
+        "name": "kubectl 命令生成",
+        "prompt": "生成 kubectl 命令來重啟 deployment api-backend。只輸出命令，不要解釋。",
+        "validators": [
+            lambda r: "kubectl" in r.lower(),  # 包含 kubectl
+            lambda r: "restart" in r.lower() or "rollout" in r.lower(),  # 包含重啟動作
+        ],
+        "description": "應生成有效 kubectl 命令",
+    },
+    {
+        "name": "風險評估",
+        "prompt": "評估風險等級：刪除 production namespace 中的 Pod。只回答 LOW/MEDIUM/HIGH/CRITICAL 其中一個。",
+        "validators": [
+            lambda r: any(level in r.upper() for level in ["HIGH", "CRITICAL"]),  # 正確識別高風險
+        ],
+        "description": "應識別為高風險操作",
+    },
+    {
+        "name": "數學推理",
+        "prompt": "計算：如果 CPU 使用率從 60% 上升到 95%，上升了多少百分點？只回答數字。",
+        "validators": [
+            lambda r: "35" in r,  # 正確答案
+        ],
+        "description": "應正確計算 35",
+    },
+]
+
+
+# =============================================================================
+# 輔助函數
+# =============================================================================
+
+async def call_ollama(prompt: str, model: str = DEFAULT_MODEL) -> str | None:
+    """呼叫 Ollama API"""
+    try:
+        async with httpx.AsyncClient(timeout=TIMEOUT) as client:
+            response = await client.post(
+                f"{OLLAMA_URL}/api/generate",
+                json={
+                    "model": model,
+                    "prompt": prompt,
+                    "stream": False,
+                },
+            )
+            response.raise_for_status()
+            return response.json().get("response", "")
+    except Exception as e:
+        print(f"Ollama 呼叫失敗: {e}")
+        return None
+
+
+async def check_ollama_available() -> bool:
+    """檢查 Ollama 是否可用"""
+    try:
+        async with httpx.AsyncClient(timeout=5) as client:
+            response = await client.get(f"{OLLAMA_URL}/api/tags")
+            return response.status_code == 200
+    except Exception:
+        return False
+
+
+# =============================================================================
+# 測試類別
+# =============================================================================
+
+class TestModelRegression:
+    """模型回歸測試"""
+
+    @pytest.fixture(autouse=True)
+    async def check_ollama(self):
+        """檢查 Ollama 可用性"""
+        available = await check_ollama_available()
+        if not available:
+            pytest.skip(f"Ollama 無法連線: {OLLAMA_URL}")
+
+    @pytest.mark.asyncio
+    @pytest.mark.parametrize("case", REGRESSION_CASES, ids=[c["name"] for c in REGRESSION_CASES])
+    async def test_regression_case(self, case: dict[str, Any]):
+        """執行回歸測試案例"""
+        response = await call_ollama(case["prompt"])
+
+        assert response is not None, f"模型無回應: {case['name']}"
+        assert len(response) > 0, f"回應為空: {case['name']}"
+
+        # 執行驗證器
+        for i, validator in enumerate(case["validators"]):
+            assert validator(response), (
+                f"驗證失敗 [{case['name']}] 驗證器 {i+1}: {case['description']}\n"
+                f"回應: {response[:200]}"
+            )
+
+
+# =============================================================================
+# 準確度報告
+# =============================================================================
+
+@pytest.mark.asyncio
+async def test_regression_report():
+    """生成回歸測試報告"""
+    available = await check_ollama_available()
+    if not available:
+        pytest.skip(f"Ollama 無法連線: {OLLAMA_URL}")
+
+    passed = 0
+    failed = 0
+    results = []
+
+    for case in REGRESSION_CASES:
+        response = await call_ollama(case["prompt"])
+
+        if response is None:
+            failed += 1
+            results.append({"name": case["name"], "status": "ERROR", "reason": "無回應"})
+            continue
+
+        all_passed = True
+        for validator in case["validators"]:
+            if not validator(response):
+                all_passed = False
+                break
+
+        if all_passed:
+            passed += 1
+            results.append({"name": case["name"], "status": "PASS"})
+        else:
+            failed += 1
+            results.append({
+                "name": case["name"],
+                "status": "FAIL",
+                "response": response[:100],
+            })
+
+    total = passed + failed
+    accuracy = (passed / total * 100) if total > 0 else 0
+
+    print("\n" + "=" * 60)
+    print("Phase 12.3: 模型回歸測試報告")
+    print("=" * 60)
+    print(f"模型: {DEFAULT_MODEL}")
+    print(f"總案例: {total}")
+    print(f"通過: {passed}")
+    print(f"失敗: {failed}")
+    print(f"準確率: {accuracy:.1f}%")
+    print("=" * 60)
+
+    if failed > 0:
+        print("\n失敗案例:")
+        for r in results:
+            if r["status"] != "PASS":
+                print(f"  - {r['name']}: {r.get('reason', r.get('response', ''))}")
+
+    # 基線門檻 75%
+    assert accuracy >= 75, f"準確率 {accuracy}% 低於基線 75%"
+
+
+if __name__ == "__main__":
+    pytest.main([__file__, "-v", "--tb=short"])
--- a/docs/LOGBOOK.md
+++ b/docs/LOGBOOK.md
@@ -5,15 +5,53 @@

 ---

-## 📍 當前狀態 (2026-03-25 14:00)
+## 📍 當前狀態 (2026-03-25 21:00)

 | 項目 | 狀態 |
 |------|------|
-| **當前 Phase** | **Phase 11 對話式 AI UI/UX** |
+| **當前 Phase** | **Phase 13 Enterprise AIOps 規劃** |
 | **Day** | Day 7 |
-| **下一步** | Phase 11.3 響應式 / 最終整合測試 |
-| **重大決策** | ✅ **Phase 11.1-11.4 完成** - 對話式容器 + 批次處理 + 鍵盤快捷鍵 |
-| **CI/CD** | ✅ Runner 恢復運作 |
+| **下一步** | Phase 13.2 Tool 實作 (P0) - SignOz/K8s/PostgreSQL MCP |
+| **重大決策** | ✅ **Phase 13 批准** - CI/CD 整合 + Tool 實作 + 智能路由 |
+| **CI/CD** | ✅ Runner 運作中，c437b0c 部署成功 |
+
+### ✅ 2026-03-25 Phase 13 Enterprise AIOps 規劃 (Day 7 晚間)
+
+**統帥架構盤點會議** - 對照業界主流 AI Agent 運用方式
+- CI/CD 結合: 符合度 60% (告警 ✅ / Git 觸發 ❌)
+- Tool 封裝: 符合度 40% (MCP 骨架 ✅ / 實際連接 ❌)
+- 智能路由: 符合度 50% (Fallback ✅ / 意圖判別 ❌)
+
+**新增 Phase 13 工作項目** (#74-88):
+- 13.1 CI/CD 整合: GitHub Webhook + AI 診斷 + 自動修復 (風險分級)
+- 13.2 Tool 實作 (P0): SignOz + K8s + PostgreSQL + RAG MCP
+- 13.3 智能路由: Intent Classifier + Complexity Scorer + AI Router
+
+**新增 Memory**:
+- feedback_tool_vs_modular.md - Tool 封裝 vs 模組化關係
+- project_phase13_enterprise_aiops.md - Phase 13 完整規劃
+
+### ✅ 2026-03-25 Phase 12.1-12.2 完成 (Day 7 下午)
+
+**Phase 12.1 Tool Calling 優化** ✅ (commit afda312, c437b0c)
+- 建立 24 個測試案例 (英/中/混合/邊界)
+- 準確率 80% → 100%
+- 新增 3 個解析模式: 中文刪除、混合重啟、明確 restart deployment
+
+**Phase 12.2 本地模型優化** ✅
+- 硬體盤點: 192.168.0.188 (62GB RAM, 無 GPU)
+- 部署 qwen2.5:7b-instruct (4.7GB)
+- Benchmark: qwen2.5 29s vs llama3.2 69s
+- ModelRouter 動態路由設計完成
+
+**Memory 新增**:
+- `project_ollama_model_inventory.md` - 模型盤點
+- `project_model_router_design.md` - 動態路由架構
+
+**Skills 更新**:
+- `03-openclaw-cognitive-expert.md` - 模型選擇規則
+
+---

 ### ✅ 2026-03-25 Phase 11 進度 (Day 7)

@@ -84,6 +122,9 @@

 | 時間 | 事件 | 負責人 |
 |------|------|--------|
+| 2026-03-25 15:30 | **🤖 Phase 12 Ollama 整合計畫批准**: Tool Calling + Kimi-K2.5 + CI/CD + Cloud Models (#60-73) | 統帥 |
+| 2026-03-25 15:20 | **✅ OpenClaw Sentry 整合提交 (4edb862)**: sentry_integration.py 已推送 gitea + GitHub | Claude Code |
+| 2026-03-25 15:00 | **🚀 Phase 11 b13b063 推送**: 對話式 AI UI/UX 全部完成，CI/CD 排隊中 | Claude Code |
 | 2026-03-25 14:00 | **🎨 Phase 11.1-11.4 完成**: ConversationalView + BatchModeSelector + useKeyboardShortcuts (Y/N 長按支援) | Claude Code |
 | 2026-03-25 11:00 | **✅ #15 SSE 改造完成 (170102a)**: Approval Polling → SSE 即時更新，新增 /api/v1/approvals/stream + useApprovalSSE hook | Claude Code |
 | 2026-03-25 10:00 | **🎨 Phase 11 對話式 AI 批准**: ChatGPT 風格 + 批次處理 + 鍵盤快捷鍵 (Y/N/方向鍵) + 響應式佈局 (#47-59) | 統帥 |