feat(aiops): enable read-only agent loop canary
All checks were successful
CD Pipeline / tests (push) Successful in 1m43s
Code Review / ai-code-review (push) Successful in 31s
CD Pipeline / build-and-deploy (push) Successful in 10m22s
CD Pipeline / post-deploy-checks (push) Successful in 4m3s

This commit is contained in:
Your Name
2026-05-01 14:20:16 +08:00
parent 33a7148916
commit f8e44971c1
9 changed files with 315 additions and 8 deletions

View File

@@ -6,6 +6,23 @@
---
## 2026-05-01 | Agent Loop P1 canary + CD Argo revision gate + SSH MCP 四節點閉環
承接 ADR-105 地基與 production 驗證後的待辦CD 會在 push deploy commit 後誤判上一個 Argo revision 已 Synced/HealthySSH MCP key 尚未授權 120/121Agent Loop 仍只停在 provider capability尚未有 production canary。
### 完成
- Gitea CD `Deploy to K8s (ArgoCD GitOps)` 在 push `chore(cd): deploy ... [skip ci]` 後,會記錄 `DEPLOY_REVISION`,先 annotate `argocd.argoproj.io/refresh=hard`,再等待 `.status.sync.revision == DEPLOY_REVISION` 且 Synced/Healthy超時直接 fail不再讓舊 revision rollout 假成功。
- `ssh-mcp-key` public key 已以一次性 privileged pod 追加到 `mon(192.168.0.120)``mon1(192.168.0.121)``wooo/.ssh/authorized_keys`;臨時 pod 已刪除。
- API pod 內使用 `/run/secrets/ssh_mcp_key` + `/etc/ssh-mcp/known_hosts` 驗證 `wooo@192.168.0.120``wooo@192.168.0.121` 均回 `OK`
- 新增 `ENABLE_OPENCLAW_AGENT_LOOP_SHADOW` / `OPENCLAW_AGENT_LOOP_MAX_ITERATIONS`production configmap 開啟 read-only canary最多 2 輪,本地 Ollama tool_use不改主決策。
- `OpenClawService.generate_incident_proposal_with_tools()` 在原 proposal 成功後執行 read-only Agent Loop shadow investigation只給 Kubernetes/Prometheus/SignOz/Database/RAG/Grafana 的 read-only MCP tools結果附加 `agent_loop_shadow` metadata。
- Agent Loop shadow 失敗只 log warning不阻塞原本 PreDecision/Nemotron/Playbook 路徑。
### 驗證
- `python3 -m py_compile apps/api/src/core/config.py apps/api/src/services/openclaw.py apps/api/src/services/ai_providers/permissions.py` 通過。
- `cd apps/api && pytest tests/test_agent_loop_foundation.py tests/test_openclaw_cache_key.py -q` → 8 passed。
- Production 前置檢查:最新 image `b6cf6167` 健康ArgoCD `Synced Healthy 33a71489`;四節點 SSH MCP key 驗證完成。
## 2026-05-01 | LLM 鬼循環治理 — in-flight lock + stable cache + no-retry 2xx
Claude Code 成本評估指出真正瓶頸不是外部 AI 費用,而是同一告警 0 秒重入、20 秒週期反覆呼叫 LLM、以及 HTTP 500 讓 Alertmanager 立即重試。結論:先修飛輪,再談 Gemini/Groq/Claude 訂閱;健康狀態下外部 provider 只應作為 capped fallback。

View File

@@ -64,6 +64,18 @@ Alert -> Agent Loop -> LLM tool_use -> MCP tool_result -> Final Decision
3. P2: OpenClaw/NemoTron/Hermes/ElephantAlpha 逐一接線,先 read-only / diagnose再開執行類工具。
4. P3: Langfuse spans、Grafana MCP dashboard、audit replay。
## 2026-05-01 P1 Canary
OpenClaw 先接 read-only shadow investigation而不是直接替換主決策
- Feature flag`ENABLE_OPENCLAW_AGENT_LOOP_SHADOW`
- 輪數上限:`OPENCLAW_AGENT_LOOP_MAX_ITERATIONS`prod canary 設 2
- 觸發點:`OpenClawService.generate_incident_proposal_with_tools()` 原 proposal 成功後
- 允許工具Kubernetes / Prometheus / SignOz / Database / RAG / Grafana 的 read-only tools
- Provider本地 Ollama 優先,不新增 Gemini/Claude 付費呼叫
- 影響面:只附加 `agent_loop_shadow` metadata不覆蓋 `action``risk_level``confidence` 或 Nemotron tool result
- 失敗策略log warning 後回到既有 proposal / Nemotron / Playbook 路徑
## 驗收
- `mcp_audit_log` 24h call count > 0。