wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
1,925 Commits 199 Branches 2 Tags
f6b698c873fa735e3be026f0ab5d0169bbdf8742
Commit Graph

326 Commits

Author SHA1 Message Date
AWOOOI CD
035fe20e4d chore(cd): deploy 0068440 [skip ci] 2026-05-03 23:45:12 +08:00
Your Name
b1ef05fa8c feat(ollama): ADR-110 GCP 三層容災架構（GCP-A → GCP-B → Local → Gemini）
Some checks failed
Code Review / ai-code-review (push) Successful in 50s
Details
CD Pipeline / tests (push) Failing after 1m14s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details 
## 變更摘要
- Primary: http://34.143.170.20:11434 (GCP-A SSD, 9x 載速 + 2x 推理)
- Secondary: http://34.21.145.224:11434 (GCP-B SSD)
- Fallback: http://192.168.0.111:11434 (M1 Pro Local HDD，最後防線)
- 廢止 ADR-105「111 唯一鐵律」，新建 ADR-110

## 核心改動
- config.py: 新增 OLLAMA_SECONDARY_URL；validator 加 GCP IP 白名單（34.143.170.20, 34.21.145.224）
- ollama_failover_manager.py: 三層 Ollama 決策矩陣；並行健康檢查三台；health_111 → health_gcp_a
- ollama_health_monitor.py: host label 萃取改為通用版（支援 GCP 公網 IP）
- failover_alerter.py: 故障/恢復主機動態顯示，不再硬編碼「Ollama 111 (GPU)」
- ollama_auto_recovery.py: notify_recovery 改為 ollama_gcp_a；recovered_host 動態
- k8s/awoooi-prod: configmap + deployment + network-policy 同步更新（egress 加 GCP /32）
- 服務層: 10 個服務檔案硬編碼 192.168.0.111 改為讀 settings.OLLAMA_URL
- 測試: URL 常數更新，新增三層容災場景，GCP IP 白名單驗證測試

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 22:49:23 +08:00
Your Name
577250a678 fix(governance): 修反消音化 W-3/W-4 守衛 + Prometheus 補資料缺失告警
Some checks failed
Code Review / ai-code-review (push) Successful in 52s
Details
CD Pipeline / tests (push) Failing after 2m21s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 1m6s
Details 
【統帥怒訓 — 違反 feedback_full_chain_first_then_fix.md 鐵律】

前次 commit f1362fcc 用 skip 條件把告警吞掉，是消音化解法：
  - W-3：total_exec<10 永遠 skip → Redis 永遠空也不會告警
  - W-4：playbooks total==0 永遠 skip → 表被清空也不會告警
  - Prometheus NaN sentinel + 既有 < 0.1 規則疊加後沒任何路徑會告警

統帥怒訓「又把告警給消失了」「已經這樣做幾次了」。本 commit 救回告警可見性。

【修法 — 啟動 30 分鐘寬限 + 過期改打資料管線斷新告警】

- ai_slo_watchdog_job.py 新增模組層 _PROCESS_START 與 _grace_active() 守衛：
  - W-3a：metric 有資料 + rate<0.30 → 既有「飛輪成功率過低」
  - W-3b：rate=None 且 uptime>30min → 新告警「飛輪資料管線無流量」
  - W-4a：playbooks total>0 + approved=0 → 既有「自動修復鏈路斷裂」
  - W-4b：playbooks total=0 且 uptime>30min → 新告警「Playbook 表初始化失敗」

- 3 份 Prometheus rule（k8s/monitoring/flywheel-alerts.yaml、
  ops/monitoring/alerts.yml、ops/monitoring/alerts-unified.yml）新增
  FlywheelExecutionRateMissing：absent() 或 NaN 持續 30 分鐘 → 告警，
  與 watchdog W-3b 雙保險

【已加入 memory】

feedback_silencing_alerts_recurring_violation.md 鎖入紅線鐵律：
  「fresh deploy / init guard 用 skip 吞告警 = 結構性失職，必須分流寬限期 +
   過期改打資料管線斷新告警」

【驗證】

106 個治理相關 unit test 全過：
  test_trust_drift_watchdog / test_governance_agent / test_failover_alerter /
  test_check_trust_drift_commit_outside_context_poc /
  test_governance_remediation_dispatch / test_ai_governance_endpoints /
  test_governance_dispatcher

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-03 12:39:46 +08:00
Your Name
dedb12085b chore(governance,watchdog): enrich alerts and enable prometheus multiproc
Some checks failed
CD Pipeline / tests (push) Failing after 1m22s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
Code Review / ai-code-review (push) Successful in 43s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 57s
Details
2026-05-02 23:44:12 +08:00
AWOOOI CD
68e182381f chore(cd): deploy da772a1 [skip ci] 2026-05-02 17:58:22 +08:00
AWOOOI CD
697e13b23a chore(cd): deploy 297afb6 [skip ci] 2026-05-02 17:28:56 +08:00
AWOOOI CD
a6409c39e2 chore(cd): deploy b3a0f0d [skip ci] 2026-05-02 16:49:00 +08:00
AWOOOI CD
329849a559 chore(cd): deploy 7795f02 [skip ci] 2026-05-01 20:53:02 +08:00
AWOOOI CD
b72eac0712 chore(cd): deploy 433f7b0 [skip ci] 2026-05-01 17:08:42 +08:00
Your Name
433f7b068e fix(aiops): close ssh and telegram remediation gaps
All checks were successful
CD Pipeline / tests (push) Successful in 2m7s
Details
Code Review / ai-code-review (push) Successful in 42s
Details
CD Pipeline / build-and-deploy (push) Successful in 13m14s
Details
CD Pipeline / post-deploy-checks (push) Successful in 4m29s
Details
2026-05-01 16:53:02 +08:00
AWOOOI CD
f72419dd17 chore(cd): deploy b0da6da [skip ci] 2026-05-01 15:27:48 +08:00
AWOOOI CD
f53d7e5584 chore(cd): deploy f8e4497 [skip ci] 2026-05-01 14:41:18 +08:00
Your Name
f8e44971c1 feat(aiops): enable read-only agent loop canary
All checks were successful
CD Pipeline / tests (push) Successful in 1m43s
Details
Code Review / ai-code-review (push) Successful in 31s
Details
CD Pipeline / build-and-deploy (push) Successful in 10m22s
Details
CD Pipeline / post-deploy-checks (push) Successful in 4m3s
Details
2026-05-01 14:20:16 +08:00
AWOOOI CD
33a7148916 chore(cd): deploy b6cf616 [skip ci] 2026-05-01 14:02:59 +08:00
AWOOOI CD
1fe75e9f99 chore(cd): deploy 6ec3f11 [skip ci] 2026-05-01 13:45:55 +08:00
Your Name
3691402561 chore(cd): deploy 11673d80 api [skip ci] 2026-05-01 12:52:23 +08:00
Your Name
11673d80ea fix(aiops): route backup decisions through ssh
Some checks failed
CD Pipeline / tests (push) Successful in 1m35s
Details
Code Review / ai-code-review (push) Successful in 34s
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
2026-05-01 12:50:01 +08:00
Your Name
ce4cf4c94b chore(cd): deploy 2c12bce api [skip ci] 2026-05-01 10:58:55 +08:00
Your Name
78bcc090ad chore(cd): deploy 97be5de api [skip ci] 2026-05-01 10:52:31 +08:00
AWOOOI CD
046d598e88 chore(cd): deploy e4aef6a [skip ci] 2026-05-01 10:43:56 +08:00
Your Name
fa6a78af2a chore(cd): deploy e4aef6a api [skip ci] 2026-05-01 10:42:07 +08:00
AWOOOI CD
7472eb2fcd chore(cd): deploy ca22ec2 [skip ci] 2026-05-01 10:24:48 +08:00
AWOOOI CD
3e0ab0f8c6 chore(cd): deploy f154ac0 [skip ci] 2026-05-01 00:14:36 +08:00
AWOOOI CD
f946e7b184 chore(cd): deploy 6e04fe9 [skip ci] 2026-04-30 23:18:20 +08:00
AWOOOI CD
64b09273f7 chore(cd): deploy e29aab5 [skip ci] 2026-04-30 15:58:18 +08:00
AWOOOI CD
a93fbe5d66 chore(cd): deploy 36967d0 [skip ci] 2026-04-30 15:44:46 +08:00
AWOOOI CD
38ffcf4395 chore(cd): deploy 712d3e5 [skip ci] 2026-04-30 15:20:33 +08:00
AWOOOI CD
ae52d51210 chore(cd): deploy 72945bf [skip ci] 2026-04-30 15:05:57 +08:00
AWOOOI CD
6e76c5dfd5 chore(cd): deploy c9393c3 [skip ci] 2026-04-30 14:41:46 +08:00
AWOOOI CD
19788302df chore(cd): deploy 80defbe [skip ci] 2026-04-30 14:26:44 +08:00
Your Name
80defbed7c fix(aiops): fallback and escalate automation blockers
Some checks failed
CD Pipeline / tests (push) Successful in 2m41s
Details
Code Review / ai-code-review (push) Successful in 24s
Details
CD Pipeline / build-and-deploy (push) Successful in 7m51s
Details
CD Pipeline / post-deploy-checks (push) Failing after 2m15s
Details
2026-04-30 14:13:57 +08:00
AWOOOI CD
9ee3cc6242 chore(cd): deploy 4723499 [skip ci] 2026-04-30 11:11:04 +08:00
AWOOOI CD
a0be4ebb03 chore(cd): deploy 0f7e9d3 [skip ci] 2026-04-30 10:54:29 +08:00
Your Name
9f15f3cfe4 chore(cd): deploy 639bb64 [skip ci] 2026-04-30 09:41:20 +08:00
AWOOOI CD
d197e2785d chore(cd): deploy 4a57c2d [skip ci] 2026-04-29 15:48:24 +00:00
AWOOOI CD
dae0aa2312 chore(cd): deploy d845d53 [skip ci] 2026-04-29 15:06:57 +00:00
AWOOOI CD
b857be0a64 chore(cd): deploy fe2b8f4 [skip ci] 2026-04-29 14:47:51 +00:00
AWOOOI CD
525a243550 chore(cd): deploy dccdcdb [skip ci] 2026-04-29 13:59:53 +00:00
AWOOOI CD
4c91d89dd2 chore(cd): deploy 4115ddd [skip ci] 2026-04-29 13:04:37 +00:00
Your Name
dc18b0ebd6 fix(prometheus_url): drift 殘存追修 — kured 守門員 + monitoring API 
debugger 全 codebase 追根溯源後揪出 5 處 PROMETHEUS_URL drift 殘存
（根因：docs/reference/SERVICE-ENDPOINTS.md 早期把 Prometheus 標在 188
是整個 codebase drift 的源頭）。

本次修最急的 2 處：

## 🔴🔴 kured.yaml:132（守門員失效風險）
- 188 → 110
- kured 跑 reboot 前會查 Prometheus alerts，連錯主機 = 跳過保護直接 reboot 主機
- 對齊 ConfigMap + config.py PROMETHEUS_URL

## 🟡 monitoring.py:67（單一事實源）
- 寫死 110:9090 改用 settings.PROMETHEUS_URL
- 主機巧合正確但繞過 ConfigMap 注入機制
- 未來 Prometheus 再遷移避免再次 drift

## 暫不修
- k3s_monitor_service.py:38 用 121:30090 是 K3s NodePort 內網端點
  與外部 PROMETHEUS_URL 概念不同，需新增 PROMETHEUS_INTERNAL_URL setting
- 其他 docstring + 文件 drift（SERVICE-ENDPOINTS.md 等）留待後續

## 驗證
1552 unit tests 全綠（無回歸）

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-29 10:44:39 +08:00
AWOOOI CD
20009cddcf chore(cd): deploy 143c15f [skip ci] 2026-04-28 07:36:19 +00:00
Your Name
143c15f052 feat(wave2+km): LLM 動態按鈕啟用 + KM 自動寫入 + AI Router dead code 標記
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 9m52s
Details 
- ConfigMap: USE_LLM_DYNAMIC_BUTTONS=true（B2/B3/B4 handler 全就緒）
- decision_manager: auto_execute 失敗路徑補 KM fire-and-forget 寫入
- ai_router: _build_fallback_chain 標記 DEPRECATED 2026-04-28
- tests: test_golden_regression.py 新增 172 行 golden 回歸測試

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-28 15:27:33 +08:00
AWOOOI CD
2e6ae7fe84 chore(cd): deploy 7f200af [skip ci] 2026-04-28 07:14:34 +00:00
AWOOOI CD
b8a330f9e4 chore(cd): deploy c1a1be6 [skip ci] 2026-04-27 12:21:13 +00:00
Your Name
c1a1be61bd fix(ssh-auto): 主機告警 SSH 自動診斷授權（HostHighCpuLoad 不再卡人工審核）
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 9m7s
Details 
根因：SSH_MCP_ALLOWED_HOSTS 未設定 → _ssh_execute() 全部攔截
      + auto_approve 只認 kubectl 不認 ssh → 主機告警永遠降級人工

修復：
- ConfigMap: 補 SSH_MCP_ALLOWED_HOSTS 四主機白名單
- alert_rules: HostHighCpuLoad 等從 NO_ACTION 改為 ssh_diagnose 指令
- auto_approve: _has_executable 加入 ssh 開頭識別
- decision_manager: _ssh_execute() 加入 ssh_diagnose 路由
- ssh_provider: 新增 ssh_diagnose tool（ps aux + free -h + df -h，只讀）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-27 20:13:07 +08:00
AWOOOI CD
ded17caca0 chore(cd): deploy a0502b7 [skip ci] 2026-04-27 11:55:33 +00:00
AWOOOI CD
0a22f49932 chore(cd): deploy e3bad58 [skip ci] 2026-04-27 08:21:06 +00:00
AWOOOI CD
dfbf3f8f20 chore(cd): deploy a184b82 [skip ci] 2026-04-27 08:08:52 +00:00
Your Name
c3fa03fc19 fix(solver): 補 AGENT_SOLVER_TIMEOUT_SEC=80 + prompt 禁無腦重啟
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
問題1：AGENT_SOLVER_TIMEOUT_SEC 預設 20s K8s 未設 → deepseek-r1:14b 必然
       timeout → candidates=[] → action="" → Telegram 顯示「待分析」+「規則分析」

問題2：Solver prompt JSON 範例只有 restart + kubectl top，LLM 模仿範例
       → 所有告警都推重啟，HostDisk/CPU 類應優先診斷+清理

修復：
- K8s 加 AGENT_SOLVER_TIMEOUT_SEC=80（< OPENCLAW_TIMEOUT=120，留 buffer）
- Solver prompt 加根因對應修復規則：HostDisk→df/du/journalctl，CPU→top/ps，
  OOM→kubectl logs，禁止「先重啟」
- JSON 範例改為 HostDisk SSH 診斷場景，不再只有 K8s 命令

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-27 15:51:42 +08:00
Your Name
1b6a4dc14c fix(k8s): 補 AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=100 救急 step_timeout
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
根因：deepseek-r1:14b 推理單題實測 28s，SRE prompt 更長必然 >30s
      AGENT_DIAGNOSTICIAN_TIMEOUT_SEC 預設 30s，K8s 沒有覆寫
      導致 diagnostician 必然 step_timeout → 信心 20% 降級

修復：K8s 加 AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=100（低於 OPENCLAW_TIMEOUT=120，留 20s buffer）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-27 15:40:46 +08:00