AWOOOI CD
|
b0bf3783e4
|
chore(cd): deploy 2c57b71 [skip ci]
|
2026-04-26 13:04:37 +00:00 |
|
Your Name
|
55c6b4e2d9
|
feat(p1): Ollama 多層容災系統 — P1.1 健康檢測 + P1.2 ai_router 整合 + P1.5 容災告警
ADR-092 P1 飛輪閉環的 Ollama 失敗轉移子系統,全部 Engineer-A2/C/C2 補上。
新服務 (1581 行):
- ollama_health_monitor.py (356):3 層健康檢測(TCP/HTTP/推理)
- ollama_failover_manager.py (571):111→188 自動切換 + Redis 持久化 + recovery callback
- ollama_auto_recovery.py (436):30s 背景監控 + 連續 3 次 HEALTHY → 切回 + clear_cache
- failover_alerter.py (218):P1.5 Telegram 容災告警
服務整合:
- ai_router.py: AIProviderEnum.OLLAMA_188 + 120s budget + failover fallback chain
- main.py lifespan: 啟動時 wire callback + start recovery,關閉時優雅 stop
- config.py: OLLAMA_FALLBACK_URL / OLLAMA_HEALTH_CHECK_MODEL / GEMINI_DAILY_QUOTA(帳單熔斷)
K8s 配置:
- 04-configmap.yaml.patch-188-fallback:注入 OLLAMA_FALLBACK_URL=http://192.168.0.188:11434
測試 (2082 行):
- test_ollama_health_monitor.py (402)
- test_ollama_failover_manager.py (707)
- test_ollama_auto_recovery.py (580)
- test_ai_router_failover_integration.py (257)
- test_lifespan_failover_wiring.py (136)
依賴鏈:service 三件套 + ai_router + main.py 一起 commit,缺一就 ImportError。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-04-26 20:18:33 +08:00 |
|
AWOOOI CD
|
4a8c3ca5c4
|
chore(cd): deploy bb12647 [skip ci]
|
2026-04-25 02:39:34 +00:00 |
|
AWOOOI CD
|
f676b61282
|
chore(cd): deploy cbd28e2 [skip ci]
|
2026-04-25 01:55:58 +00:00 |
|
AWOOOI CD
|
b8b5c68f31
|
chore(cd): deploy f9f2263 [skip ci]
|
2026-04-24 19:37:26 +00:00 |
|
AWOOOI CD
|
411a285735
|
chore(cd): deploy 250eca9 [skip ci]
|
2026-04-24 19:23:08 +00:00 |
|
Your Name
|
c14f23b33a
|
feat(k8s+notification): TG_GROUP_CUTOVER=true — 所有告警全切 SRE 群組
notification_matrix TYPE-5S: DM → GROUP(SignOz 事件補齊)
prod/dev ConfigMap TG_GROUP_CUTOVER: false → true
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-25 03:07:28 +08:00 |
|
AWOOOI CD
|
fa453fa1f3
|
chore(cd): deploy 974cc7f [skip ci]
|
2026-04-24 18:52:18 +00:00 |
|
Your Name
|
974cc7f204
|
feat(k8s): prod ConfigMap HERMES_NL_ENABLED=true
CD Pipeline / build-and-deploy (push) Successful in 13m22s
@tsenyangbot @mention 在 SRE 群組已接通,polling 路徑 → Hermes NL → 12-Agent Claude SDK
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-25 02:43:42 +08:00 |
|
AWOOOI CD
|
f48e0725e8
|
chore(cd): deploy 86ee013 [skip ci]
|
2026-04-24 18:30:57 +00:00 |
|
Your Name
|
86ee013cdf
|
feat(hermes-complete): Hermes NL 三項補強 + ConsensusEngine + ADR 收尾
CD Pipeline / build-and-deploy (push) Successful in 9m32s
## Hermes NL 補強(nl_gateway.py)
- T1 hermes_dispatch_log DB 寫入(asyncio.create_task 非阻擋)
- T2 Redis 速率限制:per-chat_id 20 req/min,fail-open
- T3 Multi-turn session:hermes:session:{chat_id}:{user_id} TTL=300s,最近 3 輪
## ConsensusEngine(ADR-095 宣告式設計)
- consensus_engine.py: CONSENSUS_WEIGHTS class 屬性
security=0.4 鎖定,9 個 Claude Code agent 分配 0.6
- config.py: ENABLE_12AGENT_CONSENSUS=False feature flag
## ADR 狀態
- ADR-093/094/095: Proposed → 🟡 批准實作中
- 各 ADR 加 v1.1 變更紀錄
## K8s ConfigMap
- prod 04-configmap.yaml: 加 3 個 feature flags(均 false)
- dev 02-configmap.yaml: 同步加入
## LOGBOOK
- 記錄 WS0–WS6 + 補強完成,feature flags 啟用指引
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-25 02:22:40 +08:00 |
|
AWOOOI CD
|
ad0e5cbbbc
|
chore(cd): deploy 0044337 [skip ci]
|
2026-04-24 18:20:09 +00:00 |
|
AWOOOI CD
|
c31bc8411f
|
chore(cd): deploy 55f111e [skip ci]
|
2026-04-24 16:21:56 +00:00 |
|
AWOOOI CD
|
6df631c895
|
chore(cd): deploy 0d81b28 [skip ci]
|
2026-04-24 16:02:18 +00:00 |
|
AWOOOI CD
|
ad494288cb
|
chore(cd): deploy c995fe4 [skip ci]
|
2026-04-24 12:49:30 +00:00 |
|
AWOOOI CD
|
8f02a9efe2
|
chore(cd): deploy 97ce5ea [skip ci]
|
2026-04-24 08:05:11 +00:00 |
|
AWOOOI CD
|
3bd105be9a
|
chore(cd): deploy 88af639 [skip ci]
|
2026-04-22 01:18:56 +00:00 |
|
AWOOOI CD
|
757a58cc60
|
chore(cd): deploy 1625e7b [skip ci]
|
2026-04-21 18:10:42 +00:00 |
|
AWOOOI CD
|
ca8361e0bc
|
chore(cd): deploy 6d5f070 [skip ci]
|
2026-04-21 17:56:34 +00:00 |
|
Your Name
|
d0591c54b0
|
fix(security): 體健修復 — 7項 Critical/Major 安全問題全修
CD Pipeline / build-and-deploy (push) Failing after 35s
## Critical 修復 (C1-C5)
- C1: git rm --cached 03-secrets.yaml(CHANGE_ME 模板不再追蹤)
- C2: git rm --cached awoooi.db + .gitignore 加 *.db(SQLite HARD_RULES 違規)
- C3: sentry-tunnel SENTRY_HOST 改為 process.env fallback
- C4: config.py DATABASE_URL 移除 changeme default,改為必填
- C5: run_migration.py 改為 os.environ["DATABASE_URL"]
## Major 修復 (M1-M4)
- M1: auto_repair /execute 加 CSRF 保護 + AutoRepairPanel.tsx 同步
- M2: drift /rollback /adopt 加 CSRF 保護(/internal/scan 保持無 CSRF)
- M3: terminal /intent 加 CSRF 保護 + terminal.store.ts 同步
- M4: live-dashboard HOST_IPS + host-grid VIP 改為 env var
## 其他
- 新增 apps/web/.env.example(6 個 env var 說明)
- K8s deployment-web 補入 3 個新 env var
- 整合測試:新增 aider_event_repository + ai_router_feedback 真實 DB 測試
- test_terminal.py CSRF dependency override 修復
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
2026-04-22 01:27:39 +08:00 |
|
AWOOOI CD
|
49e465954c
|
chore(cd): deploy 4fc1f49 [skip ci]
|
2026-04-21 14:35:32 +00:00 |
|
AWOOOI CD
|
0a72ae21e4
|
chore(cd): deploy 8fd31ec [skip ci]
|
2026-04-21 13:38:44 +00:00 |
|
AWOOOI CD
|
4bc183742f
|
chore(cd): deploy bd73548 [skip ci]
|
2026-04-21 13:26:51 +00:00 |
|
AWOOOI CD
|
a2777aee04
|
chore(cd): deploy 685f5c6 [skip ci]
|
2026-04-21 13:05:41 +00:00 |
|
AWOOOI CD
|
4bc52a9bdc
|
chore(cd): deploy acab1cd [skip ci]
|
2026-04-21 07:29:25 +00:00 |
|
AWOOOI CD
|
3c266190cf
|
chore(cd): deploy 3323a90 [skip ci]
|
2026-04-20 17:13:47 +00:00 |
|
AWOOOI CD
|
e60c064bdc
|
chore(cd): deploy 9a44516 [skip ci]
|
2026-04-20 12:29:49 +00:00 |
|
AWOOOI CD
|
f9ff23f007
|
chore(cd): deploy 156a52f [skip ci]
|
2026-04-20 12:09:31 +00:00 |
|
AWOOOI CD
|
72aea671b3
|
chore(cd): deploy ce918ee [skip ci]
|
2026-04-20 11:48:59 +00:00 |
|
AWOOOI CD
|
770e869f7e
|
chore(cd): deploy 803b389 [skip ci]
|
2026-04-19 20:31:09 +00:00 |
|
AWOOOI CD
|
525102d87e
|
chore(cd): deploy 4188df6 [skip ci]
|
2026-04-19 20:22:13 +00:00 |
|
AWOOOI CD
|
a837172fd5
|
chore(cd): deploy f572561 [skip ci]
|
2026-04-19 15:10:19 +00:00 |
|
AWOOOI CD
|
b9068d495f
|
chore(cd): deploy fa643eb [skip ci]
|
2026-04-19 14:47:23 +00:00 |
|
AWOOOI CD
|
2af623032a
|
chore(cd): deploy 37b6c9b [skip ci]
|
2026-04-19 14:31:48 +00:00 |
|
AWOOOI CD
|
b9c4896c7f
|
chore(cd): deploy 2f5cab2 [skip ci]
|
2026-04-19 14:10:25 +00:00 |
|
AWOOOI CD
|
32959db83d
|
chore(cd): deploy 0004554 [skip ci]
|
2026-04-19 13:29:28 +00:00 |
|
AWOOOI CD
|
f1b13d7b26
|
chore(cd): deploy 7db8845 [skip ci]
|
2026-04-19 12:36:04 +00:00 |
|
AWOOOI CD
|
638053346b
|
chore(cd): deploy ceb61c3 [skip ci]
|
2026-04-19 12:15:43 +00:00 |
|
AWOOOI CD
|
576f9dad18
|
chore(cd): deploy ba18ad2 [skip ci]
|
2026-04-19 11:46:35 +00:00 |
|
AWOOOI CD
|
e84338e615
|
chore(cd): deploy 6ab0ce9 [skip ci]
|
2026-04-19 10:18:43 +00:00 |
|
AWOOOI CD
|
691bdc6cc1
|
chore(cd): deploy e677773 [skip ci]
|
2026-04-19 09:35:27 +00:00 |
|
AWOOOI CD
|
46677a3392
|
chore(cd): deploy df71c9a [skip ci]
|
2026-04-19 09:12:54 +00:00 |
|
AWOOOI CD
|
0d2455ae9a
|
chore(cd): deploy fdf8b73 [skip ci]
|
2026-04-19 09:01:49 +00:00 |
|
AWOOOI CD
|
c77ce63a32
|
chore(cd): deploy 0226344 [skip ci]
|
2026-04-19 08:39:23 +00:00 |
|
AWOOOI CD
|
2dd02bec3f
|
chore(cd): deploy 5b9b36f [skip ci]
|
2026-04-19 08:18:49 +00:00 |
|
OG T
|
7e4d83e66e
|
chore(cd): manual deploy e7ba8cb (CI B5 network bug bypass) [skip ci]
CI B5 Integration Tests 因 docker network 隔離無法連 pg-test-b5,
連續 2 次 fail (run 984 + 985)。
905 unit test + 26 verifier test 全 pass,確認 e7ba8cb 程式碼正確。
手動 build linux/amd64 image 推 Harbor,改 kustomization.yaml 觸發 ArgoCD sync。
下一輪需修 CI: cd.yaml B5 step 加 --network 讓 pg-test-b5 與 ci-runner 同 network。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-04-19 12:46:36 +08:00 |
|
AWOOOI CD
|
da7956187e
|
chore(cd): deploy 2abc91e [skip ci]
|
2026-04-19 03:38:47 +00:00 |
|
OG T
|
eab3f527cd
|
feat(monitoring): Phase 7 盲區治理 — L2 配額 + 自監控告警 (ADR-090)
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 1m21s
CD Pipeline / build-and-deploy (push) Failing after 9m24s
戰場:110 load=17 持續 13 天 + 188 cadvisor 321% CPU 重啟無效
統帥鐵律:不要只降低,要長期解決 → 結構性治理而非補丁
本 commit 涵蓋:
1. k8s/monitoring/docker-compose-110.yml
- cadvisor 加 mem_limit 512M + cpus 1.0(L2 防爆網)
- 備註 110 live 與本檔 drift(下一 session 納入 CD)
2. ops/monitoring/alerts-unified.yml 新增 infra_self_monitoring 群組:
- CadvisorDown / MemoryPressure / CPUThrottled
- NodeExporterDown / CPUThrottled
- SentryClickHouseMemoryPressure / CPUThrottled
- GiteaMemoryPressure / CPUThrottled
- PrometheusDown(監控自監控元層)
→ 全部用 (memory usage / spec_memory_limit) 動態判斷,
不寫死 80% 或 MB 數,配額改閾值自動跟著變
其他配套(非本 repo,已 SSH patch 到 110/188):
- /home/ollama/wooo-aiops/docker-compose.yml:188 cadvisor 加 --disable_metrics / --docker_only / --housekeeping_interval + 1g/1.5c
- /home/wooo/monitoring/docker-compose.yml:110 cadvisor + node-exporter 納管 + 降維 flags + 配額
- /opt/sentry/docker-compose.override.yml:Sentry L2 配額(clickhouse 8g/4c, kafka 3g/2c 等)
- /home/wooo/gitea/docker-compose.yml:Gitea 3g/3c
- /home/wooo/act-runner/docker-compose.yml:Actions Runner 2g/2c
對映:
- feedback_monitor_self_monitoring.md 🔴🔴🔴 監控工具必須被監控
- feedback_ai_autonomous_direction.md 動態閾值 ≠ 寫死規則
- ADR-090 Layer 2 資源配額強制
驗收(48h):
- 188 cadvisor CPU 從 321% → <50%(配額強制)
- 110 load5 從 18 → <10(Sentry/Gitea 釋壓後)
- 自監控告警無誤報
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-04-19 01:50:41 +08:00 |
|
AWOOOI CD
|
be76100112
|
chore(cd): deploy 4b8be32 [skip ci]
|
2026-04-18 17:26:35 +00:00 |
|
AWOOOI CD
|
41e6b503e2
|
chore(cd): deploy 98aef55 [skip ci]
|
2026-04-18 16:11:01 +00:00 |
|