wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity
1,261 Commits 199 Branches 2 Tags
c1c96ab47b3828c40286b4edd4e2de1a1f2eaaa5
Commit Graph

128 Commits

Author SHA1 Message Date
OG T
c1c96ab47b feat(m4): ADR-074 M4 — 備份還原週排程驗證 CronJob 
- scripts/cron_backup_restore_test.sh: Velero restore dry-run 腳本
- k8s/awoooi-prod/16-cronjob-backup-restore-test.yaml: 每週日 02:00 台北執行
- k8s/awoooi-prod/17-configmap-backup-restore-scripts.yaml: 腳本 ConfigMap
- flywheel-alerts.yaml: BackupRestoreTestFailed + BackupRestoreTestStale 告警

失敗時寫入 node-exporter textfile → Prometheus 告警 → TYPE-3 Incident

2026-04-12 ogt (ADR-074 M4)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:36:30 +08:00
OG T
00a31abb85 feat(heartbeat): ADR-073 P2 心跳整合重構 — HeartbeatReportService + RedisLock
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- 新增 HeartbeatReportService：11 個並行探針（Ollama/Nemotron/Gemini/Claude/MCP×4/ArgoCD/Velero）
- 重寫 send_heartbeat()：RedisLock 防重發 + 統一發送 SRE_GROUP_CHAT_ID
- 簡化 _heartbeat_loop()：移除散落的 silence 多次發送
- config.py：新增 OLLAMA_REQUIRED_MODELS 欄位
- 03-secrets.example.yaml：補 SRE_GROUP_CHAT_ID 確保 CD Inject 不遺漏

2026-04-12 ogt (ADR-073 Phase 2-3/4)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:35:13 +08:00
OG T
16d682346a feat(adr-074): M1 飛輪健康度 Exporter + M2 主機網路監控
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
ADR-074 M1:
  - FlywheelStatsService: 計算6項飛輪指標（Playbook數/成功率/KM向量化/alertname NULL/卡住數）
  - GET /api/v1/stats/flywheel — 六節點即時狀態（C1 前端用）
  - GET /api/v1/stats/summary — KPI 面板數據（C1 前端用）
  - GET /api/v1/stats/flywheel/metrics — Prometheus text format
  - flywheel-alerts.yaml: 5條告警規則（FlywheelPlaybookZero/ExecutionSuccessLow/KMVectorizationLow/AlertnameNullHigh/IncidentsStuck）
  - prometheus.yml: awoooi-flywheel scrape job（5分鐘間隔）

ADR-074 M2:
  - prometheus.yml: host-connectivity Blackbox TCP probe（110:22/188:22/120:6443/121:6443）
  - flywheel-alerts.yaml: HostNetworkPartition 告警規則

597 unit tests passed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:31:01 +08:00
OG T
0d239838b4 fix(cr): Code Review P2 — 測試覆蓋 + CronJob 腳本重構
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
P2-1: CronJob inline Python 抽成 scripts/cron_km_vectorize.py
      Dockerfile 加入 COPY scripts/，CronJob YAML 改用腳本路徑
P2-2: 新增 test_classify_alert_early.py — 23 tests 覆蓋 7 條分類規則
      含邊界情況：VeleroBackupFailed(backup優先於k8s)、優先順序驗證

595 unit tests passed

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 15:14:44 +08:00
OG T
f2fc4712ad feat(flywheel): Phase 4 — KM conversion hook + daily vectorize cron
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
ADR-073 Phase 4-2: incident_service.resolve_incident() KM conversion hook
- resolve 時 fire-and-forget KMConversionService.convert(incident)
- 已解決的 Incident 自動轉換為結構化 KM 條目，完成飛輪「學習固化」節點
- KMConversionService (Phase 4-1) 已存在 (km_conversion_service.py, 336 lines)

ADR-073 Phase 4-3: 15-cronjob-km-vectorize.yaml
- 每日 03:00 台北時間呼叫 /api/v1/knowledge/embed-all
- 自動向量化當日新增 KM 條目，確保 RAG 查詢不遺漏新知識
- 加入 kustomization.yaml resources

Phase 4-4: handle_callback log_manual_fix 已存在 (telegram_gateway.py:2468)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:40:33 +08:00
AWOOOI CD
59b7d8ea32 chore(cd): deploy 6dc03c9 [skip ci] 2026-04-12 06:30:20 +00:00
OG T
6dc03c9a55 fix(argocd)+feat(flywheel): Phase 1 完成 — ArgoCD image 斷路修復 + 冷啟動腳本
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
1. k8s/argocd/awoooi-prod-app.yaml:
   移除 Deployment image ignoreDifferences
   - 原設計造成 CD 更新 kustomization.yaml 後 ArgoCD 不更新 image
   - 修復後 GitOps 閉環恢復正常

2. scripts/cold_start_playbooks.py:
   ADR-073 Phase 1 Step 8 — 生成 15 個基礎 Playbook (K8s/Docker/DB/Infra)
   執行結果: Playbooks 0 → 15

3. scripts/batch_vectorize_km.py:
   ADR-073 Phase 1 Step 9 — 批次向量化 KM
   執行結果: 711/713 embedding IS NOT NULL

Phase 1 全部完成，飛輪已解封：
- Pod 運行 105998d（含 8be87b0 所有修復）
- debounce 30min + alertname NULL 修復 + _collect_mcp_context 啟用
- 15 Playbooks + 711 KM 向量化

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 14:20:52 +08:00
AWOOOI CD
b3fdabeb91 chore(cd): deploy 105998d [skip ci] 2026-04-12 06:06:03 +00:00
OG T
7c4b36c2cd fix(flywheel): Phase 1 — 部署 8be87b0 + debounce 30min + alertname NULL 修復
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m29s
Details 
ADR-073 Phase 1 三項修復:

1. k8s/kustomization.yaml: newTag a86ecf3 → 8be87b0
   - 解封 _collect_mcp_context + auto_approve + DESTRUCTIVE_PATTERNS
   - 這是飛輪解封的關鍵

2. webhooks.py: DEBOUNCE_WINDOW_MINUTES 5 → 30
   - 防止同一問題每 5 分鐘重建 Incident，改為 30 分鐘收斂窗口

3. incident_repository.py: signals JSONB 補充 alertname key
   - signal.model_dump() 只有 alert_name，DB query 用 signals->0->>'alertname'
   - 補充 alertname alias，修復 132 筆 incidents.alertname = NULL 根因

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 13:41:22 +08:00
AWOOOI CD
fdb8c2b97b chore(cd): deploy a86ecf3 [skip ci] 2026-04-12 04:28:38 +00:00
OG T
a86ecf32a2 fix(cd): 修復 non-fast-forward push 失敗 + 部署 8be87b0 修復版
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 19m9s
Details 
1. kustomization.yaml: c439277 → 8be87b0 (auto_approve/decision_manager/webhooks)
2. cd.yaml: git push 前先 fetch+rebase，避免 CI 期間其他 commit 造成 non-fast-forward

8be87b0 包含:
- auto_approve: high risk 開放自動執行 + DESTRUCTIVE_PATTERNS 攔截
- decision_manager: classify_notification() 接通 + NO_ACTION 早退 + MCP context 收集
- webhooks: target_resource 修正 (name/container label 提取，DockerContainerUnhealthy 不再 target=alertname)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-12 12:17:02 +08:00
AWOOOI CD
45cf1b869f chore(cd): deploy c439277 [skip ci] 2026-04-11 14:04:07 +00:00
AWOOOI CD
33a6f34104 chore(cd): deploy 615822d [skip ci] 2026-04-11 13:29:38 +00:00
AWOOOI CD
37dfbaf26c chore(cd): deploy f23176c [skip ci] 2026-04-11 13:19:04 +00:00
OG T
f23176cbb9 fix(k8s): ArgoCD MCP 網路連線修復 — ARGOCD_URL 改用 120:30443
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details 
- NetworkPolicy v1.4: 加入 ArgoCD MCP egress 規則
  - argocd namespace Pod selector (port 8080, ClusterIP fallback)
  - 192.168.0.120:30443 NodePort（ClusterIP DNAT 跨 namespace 不穩定）
- ARGOCD_URL: 192.168.0.125 → 192.168.0.120:30443（K3s Master NodePort，更穩定）
- 已驗證: 192.168.0.120:30443 從 Pod 內部可達，apps=[awoooi-prod]

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 21:10:52 +08:00
AWOOOI CD
4a00573a20 chore(cd): deploy 4b591d1 [skip ci] 2026-04-11 13:07:59 +00:00
OG T
4b591d130f chore: ArgoCD MCP egress NetworkPolicy + LOGBOOK Session 6
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- k8s NetworkPolicy v1.4: 新增 argocd namespace egress (port 80/443)
- LOGBOOK: Session 6 審計條目

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 20:59:25 +08:00
AWOOOI CD
59dff1a478 chore(cd): deploy f2c18c4 [skip ci] 2026-04-11 12:54:21 +00:00
AWOOOI CD
694471891f chore(cd): deploy 82e1c05 [skip ci] 2026-04-11 12:45:05 +00:00
AWOOOI CD
2ad2a7ba45 chore(cd): deploy f323633 [skip ci] 2026-04-11 12:18:44 +00:00
OG T
b5aa607a30 fix(mcp): 修正 Prometheus URL (110:9090) + Sentry DSN 改 HTTP 內網
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m45s
Details 
- PROMETHEUS_URL: 188:9090 → 110:9090 (Prometheus server 正確位置)
- SENTRY_DSN: https://sentry.wooo.work → http://192.168.0.110:9000 (消除 SSL hostname mismatch)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 19:51:20 +08:00
OG T
95b61802be fix(mcp): ssh-mcp-key volumeMount 路徑修正 — subPath 對齊 ssh_provider.py
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 7m45s
Details 
- ssh_mcp_key → /run/secrets/ssh_mcp_key (SSH_KEY_PATH)
- known_hosts → /etc/ssh-mcp/known_hosts (SSH_MCP_KNOWN_HOSTS_FILE)

同步: K8s Secret 重建（含 ssh_mcp_key + known_hosts）
      188/110 authorized_keys 已加入公鑰
      SSH 連線驗證: 188 OK / 110 OK

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:59:29 +08:00
OG T
b1c1091787 feat(mcp): MCP Phase 2a — SSH MCP key volume + SSH/ArgoCD/Sentry MCP 啟用
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 7m58s
Details 
- 06-deployment-api.yaml: ssh-mcp-key volume 定義（optional: true, 0400）
- 04-configmap.yaml: SSH_MCP_ENABLED/KNOWN_HOSTS_FILE + ARGOCD_MCP_ENABLED + SENTRY_MCP_ENABLED

MCP Phase 1-4 全部實作完成，10 providers 全部已啟用（ArgoCD/Sentry/SSH 需人工 Secret）

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 09:35:52 +08:00
OG T
7f4ec717ef feat(gitops): Sprint B-2/B-3 — ArgoCD Application + CD GitOps 模式 
B-2: k8s/argocd/awoooi-prod-app.yaml
  - ArgoCD Application awoooi-prod 建立（已 apply 到 K8s）
  - automated sync: prune + selfHeal
  - ignoreDifferences: Deployment image + Secret data
  - 全部 17 個 K8s 資源已確認 Synced

B-3: .gitea/workflows/cd.yaml — Deploy step 重寫
  - 舊: kubectl set image（與 ArgoCD selfHeal 衝突）
  - 新: kustomize edit set image → git commit [skip ci] → push → ArgoCD sync
  - 新增等待 ArgoCD Synced + Healthy（最多 120s）
  - 需建立 Gitea Secret: GITEA_CD_TOKEN（見 ADR-069）

docs/adr/ADR-069-infra-gitops-sprint-b.md
  - 決策記錄：循環觸發防護 + ignoreDifferences 設計

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:57:42 +08:00
OG T
6351e9a0e9 feat(mcp-phase2): MCP Phase 2 — Prometheus MCP + SSH MCP + alert labels
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 13m37s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 35s
Details 
MCP-2b: prometheus_provider.py
  - prometheus_query (PromQL 即時查詢)
  - prometheus_query_range (歷史趨勢，預設 15 分鐘)
  - prometheus_get_alert_history (告警觸發歷史)
  - config: PROMETHEUS_URL + PROMETHEUS_MCP_ENABLED

MCP-2a: ssh_provider.py
  - 群組A 9 個只讀診斷工具 (top/disk/memory/logs/status/port/nginx/swap)
  - 群組B 6 個安全操作工具 (restart/compose/systemctl/clear-log/ssl/nginx-reload)
  - 四層安全守衛 (白名單/allowed_hosts/forbidden_patterns/trust_score)
  - config: SSH_MCP_ENABLED + SSH_MCP_ALLOWED_HOSTS

K8s: 04-ssh-mcp-secret.example.yaml (ssh-mcp-key Secret 範本 + 建立步驟)

Alert labels: alerts-unified.yml 補充 mcp_provider/host_type/alert_category
  覆蓋: HostHighCpuLoad/HostOutOfMemory/HostOutOfDiskSpace/DockerContainer*
        SignOzDown/SentryDown/HarborDown/GiteaDown

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:35:35 +08:00
OG T
45b13f1d7c fix(k8s): 更新 03-secrets.example.yaml — Sentry DSN 改 HTTPS 公網域名 
ADR-069 Sprint A A-0-5:
- SENTRY_DSN: http://...@192.168.0.110:9000/3 → https://...@sentry.wooo.work/3
- 同步 Web DSN 範例（NEXT_PUBLIC_SENTRY_DSN）
- 加入取得 DSN 的步驟說明
- system.url-prefix 已設定為 https://sentry.wooo.work

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-11 02:05:11 +08:00
OG T
dbb8104557 fix(drift): kubectl not found + RBAC services/configmaps/ingresses
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
drift_detector 用 kubectl 比對 Git YAML vs K8s 實際狀態，但：
1. API image 沒有 kubectl binary → No such file or directory: 'kubectl'
2. awoooi-executor ClusterRole 缺少 services/configmaps/ingresses list 權限

修復:
- Dockerfile: apt install curl + download kubectl v1.29.0 amd64
- 07-rbac.yaml: 加入 services/configmaps (core) + ingresses (networking.k8s.io) get/list/watch

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-10 00:49:56 +08:00
OG T
62cb274735 feat(host_aggregator+k8s): 新增 121 K3s Worker 主機監控
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
HOST_CONFIGS 加入 192.168.0.121(K3s Worker):
- K3s API tcp:6443
- awoooi-api NodePort tcp:32334
- awoooi-web NodePort tcp:32335

NetworkPolicy 補開 121 egress: 6443/32334/32335
NodePort 服務實際在 121(mon1)，非 120(mon)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 23:36:36 +08:00
OG T
c92cdeea0f feat(drift): B4 drift_reports DB 持久化 + CronJob 修復
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m17s
Details 
- drift_repository.py: DriftReportRepository (save/get/list/update)
- drift.py router: 移除 in-memory dict，改用 DB repository
- drift-cronjob.yaml: 修正 SA/NetworkPolicy/NodePort 問題
- allow-intra-namespace NetworkPolicy (已套用至 prod)
- migrate-phase8/9: symptoms_hash + drift_reports migration Job YAML

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 20:28:55 +08:00
OG T
c200d7a52d fix(web+k8s): CSRF mismatch + NetworkPolicy 缺少監控端口
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m19s
Details 
1. pending-approvals-card: 改為點擊時即時 fetch 新 CSRF token
   避免多 useCSRF 實例互相覆蓋 cookie 導致 header/cookie 不一致
2. NetworkPolicy: 補開 110:3002(Grafana) 9090(Prometheus) 3001(Gitea)
   修正 monitoring probe "All connection attempts failed"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 20:11:00 +08:00
OG T
34f0228d92 fix(executor): K8s ClusterIP 10.43.0.1 不可達 — 加 K8S_API_SERVER_URL 覆蓋 + migration job
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m0s
Details 
問題: in-cluster config 讀到 10.43.0.1:443，但 K3s Pod 內 iptables/kube-proxy
      沒把流量導到實際 API server，導致 Connection refused，批准後 kubectl 永遠失敗

修復:
- executor.py: load_incluster_config() 後讀 K8S_API_SERVER_URL env 覆蓋 host
- 04-configmap.yaml: 設 K8S_API_SERVER_URL=https://192.168.0.120:6443
- migrate-sprint5r-telegram-message-id.yaml: approval_records 新增兩欄 migration job

E2E 驗證: kubectl rollout restart deployment/awoooi-worker success=True ✅

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 19:10:27 +08:00
OG T
a4d6b3f3e6 fix(review): 首席架構師+QA 修復 C1/P1/P2/I2/I3 — Sprint 5R Review 修正
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
C1/P1-1: DB migration — approval_records 新增 telegram_message_id/telegram_chat_id
  - apps/api/migrations/sprint5r_telegram_message_id.sql (新增)
  - apps/api/src/db/base.py: init_db() 加 ALTER TABLE ADD COLUMN IF NOT EXISTS
  - k8s/jobs/migrate-sprint5r-telegram-message-id.yaml (追蹤)

P1-2: risk_map 補 "high" 鍵防止 LLM 回傳 high 時降為 MEDIUM
  - apps/api/src/services/proposal_service.py:183

I2/M3: kubectl_command 回填補齊 delete_deployment/drain_node/cordon_node/delete_service
       + 抽取 _backfill_kubectl_command() helper 消除重複邏輯
  - apps/api/src/services/openclaw.py

I3: _notify_approval_result 靜默 except 改為 logger.warning
  - apps/api/src/services/telegram_gateway.py

P2-2: PendingApprovalsCard 審批動作加 loading/disabled 防止重複點擊
  - apps/web/src/components/shared/pending-approvals-card.tsx

P2-3: SecurityPanel/CompliancePanel error 死碼修復 — catch() 補 setError()
  - apps/web/src/components/panels/SecurityPanel.tsx (含 'Unresolved' i18n)
  - apps/web/src/components/panels/CompliancePanel.tsx

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 18:38:10 +08:00
OG T
07a097c259 fix(infra): Sprint 3 自動修復全鏈路修復 — docker-188 SSH egress + service registry 擴充
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
NetworkPolicy: 新增 192.168.0.188:22 egress — repair-bot-188.sh 執行路徑
service-registry.yaml: 新增 signoz/bitan-app (AUTO, 188主機)

修復覆蓋: Bug #11 補完 (188 SSH) + 188 服務分級覆蓋
E2E 驗證: MoWoooWorkDown → SSH → REPAIR_OK:momo-app (3791ms)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 18:04:19 +08:00
OG T
77f2da9264 fix(k8s): Bug #11+#12 — SSH egress 白名單 + repair-ssh-key 讀取權限
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
Bug #11 (NetworkPolicy): allow-required-egress 缺少 192.168.0.110:22
  - K8s Pod 到 110 的 SSH port 22 被 default-deny-all 封鎖
  - 自動修復的 SSH_COMMAND Playbook 必然 Connection refused
  - 修正: 加入 port 22 到 110 的 egress 白名單

Bug #12 (Deployment): repair-ssh-key Secret defaultMode=0400 (root-only)
  - Pod 以 appuser(UID 1000) 跑，無法讀取 root-owned 的 SSH key
  - ssh 報錯: "Load key: Permission denied"
  - 修正: 加入 securityContext.fsGroup=1000，讓 appuser 透過 group read 存取
  - 已驗證: Pod 內 ssh → repair-bot-110 → REPAIR_OK:sentry ✅

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 14:50:49 +08:00
OG T
08db3580a7 fix(monitoring): 修復 110 主機 CPU 高負載
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details 
根因 1: cadvisor 持續掃描 overlay2 磁碟用量 (每次 1-4s × N 容器)
  → 加 --disable_metrics=disk,diskIO,tcp,udp,percpu,sched,process
  → --housekeeping_interval=30s --docker_only=true
  → CPU 從 239% 降到 <1%

根因 2: node_exporter scrape_timeout 預設 10s，高 load 下超時→broken pipe→瘋狂重試
  → 加 scrape_interval: 30s / scrape_timeout: 25s
  → CPU 從 48% 降到 0%

整體 load average: 20 → 9

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 13:53:13 +08:00
OG T
5bd8a8a719 fix(monitoring): 補齊 blackbox-tcp scrape targets (11→15)
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
SentryDown/HarborDown/SignOzDown 等告警引用的 instance 不在 scrape list
導致 absent metric = 0，告警持續 firing

新增缺少的 targets:
  192.168.0.125:6443/32334/32335 (K3s)
  192.168.0.110:9000/5000/3100 (Sentry/Harbor/Langfuse)
  192.168.0.188:3301/5432/6380/11434/8089 (SignOz/PG/Redis/Ollama/OpenClaw)

已在 110 主機 reload Prometheus，全部 15 targets UP

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 12:20:19 +08:00
OG T
85d4857d1b fix(monitoring): RedisMemoryHigh 誤報 — max_bytes=0 除以零修正
Some checks are pending
CD Pipeline / build-and-deploy (push) Has started running
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 37s
Details 
- 加入 redis_memory_max_bytes > 0 前置條件
- 防止 Redis 未設 maxmemory 時除以零產生 +Inf 永遠觸發
- 影響: alerts-unified.yml + database-alerts.yaml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-09 11:41:10 +08:00
OG T
4aa7c179c1 feat(k8s): Sprint 5.1 Guardrail — service-registry ConfigMap 掛載到 API 容器
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 16m36s
Details 
問題: Docker 容器無 ops/ 目錄，service_registry.py 找不到 YAML → 全部降級 AUTO
解法: ConfigMap 掛載 service-registry.yaml 到 /app/ops/config/

變更:
- k8s/awoooi-prod/15-service-registry-configmap.yaml (新增 ConfigMap)
- k8s/awoooi-prod/06-deployment-api.yaml (volumeMount + volume)
- .gitea/workflows/cd.yaml (Step 1c apply ConfigMap)

效果: _find_registry_path() 可找到 YAML → BLOCK/CRITICAL_HITL/STANDARD_HITL 生效

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 22:12:29 +08:00
OG T
8b5db2f58e feat(infra): 切換 Ollama 到 M1 Pro 192.168.0.111 + NetworkPolicy 更新
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- OLLAMA_URL: 188 → 111 (M1 Pro, 40+ tok/s vs 0.45 tok/s)
- OPENCLAW_DEFAULT_MODEL: qwen2.5:7b-instruct → deepseek-r1:14b (SRE最強推理)
- OPENCLAW_TIMEOUT: 90s → 120s (deepseek-r1:14b 實測最慢 54s)
- NetworkPolicy v1.3: 新增 192.168.0.111:11434 egress，移除 188 的 Ollama port

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 22:05:14 +08:00
OG T
88696dba9b feat(sprint5.1): Data Safety Guardrails 全鏈路整合 (L1-L5)
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m33s
Details
Type Sync Check / check-type-sync (push) Failing after 58s
Details 
Layer 0 - K8s RBAC:
  - k8s/rbac/api-velero-reader.yaml: awoooi-executor SA Velero backup reader

Layer 1 - DB Migration (已在 188 執行):
  - M-002: approval_records 新增 approval_level/votes/required_votes
  - M-003: alert_event_type ENUM 新增 8 個值

Layer 2 - IaC:
  - ops/config/service-registry.yaml: 全服務 Stateful 分級清單 (BLOCK/CRITICAL_HITL/STANDARD_HITL/AUTO)

Layer 3 - Python Services:
  - service_registry.py: 讀取 YAML，提供 is_blocked/requires_multisig/get_required_votes
  - velero_client.py: kubectl 查詢 Velero 備份年齡，失敗 fallback 999h
  - preflight_service.py: Pre-flight 安全檢查 (Q2/Q4 決策)

Layer 1-M001 - Playbook model:
  - playbook.py: 新增 requires_approval_level/stateful_targets/requires_pre_backup

Layer 4 - 業務邏輯:
  - alert_operation_log_repository.py: 新增 8 個 event_type (Guardrail/Pre-flight/MultiSig/備份)
  - auto_repair_service.py: 注入 Service Registry Guardrail 檢查 (BLOCK → 直接拒絕)
  - webhooks.py: ALERT_RECEIVED 溯源記錄 + auto_repair flag Q9 + Langfuse trace_id Q10
  - db/models.py: ApprovalRecord 同步 approval_level/votes/required_votes 欄位
  - docker-health-monitor.sh: 純感知層改造（移除所有 docker restart 邏輯）

Layer 5 - Telegram 通知:
  - telegram_gateway.py: T1-T6 六個新通知方法 (Guardrail/Pre-flight/Backup/MultiSig/ChangeApplied)

參考: ADR-062 Data Safety Guardrails, ADR-063 Service Registry IaC

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-08 16:24:09 +08:00
OG T
af07c23675 fix(k8s): known_hosts 改掛 /etc/repair-known-hosts 獨立目錄，修 mount 衝突
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m11s
Details
E2E Health Check / e2e-health (push) Successful in 34s
Details 
/etc/repair-ssh 已被 repair-ssh-key 佔用，subPath 檔案掛載衝突
改為獨立目錄 /etc/repair-known-hosts，路徑同步更新 KNOWN_HOSTS_PATH

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-06 15:06:28 +08:00
OG T
d56aae135d fix(k8s): repair-known-hosts secret optional:true — Pod 不阻塞等待 secret 建立
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m35s
Details 
CD 首次跑時才建立 secret，optional 讓 Pod 先起來
等 CD 建立 secret 後自動掛載

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-06 14:48:45 +08:00
OG T
d4cb9a4ac5 ops(k8s): known_hosts Secret + Ansible 白名單 ConfigMap (Sprint 3 T2) 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-06 14:20:14 +08:00
OG T
23364423fa feat(webhook): ADR-059 GitHub → Gitea Webhook 遷移完成 
- gitea_webhook.py: Header 全部改 X-Gitea-*，移除 workflow_run handler
- gitea_webhook_service.py: _fetch_pr_diff 改直接 httpx，不依賴 github_api_service
- 清除兩個檔案的所有殘留 github_ log key，review_id prefix 改 gitea-
- test_gitea_webhook.py: 10/10 通過，docstring 修正
- 03-secrets.yaml: 新增 GITEA_WEBHOOK_SECRET 佔位
- cd.yaml: 新增 GITEA_WEBHOOK_SECRET 注入步驟
- ADR-059: 建立架構決策文件

待統帥操作: Gitea Actions secret + Gitea UI Webhook 設定

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 14:44:32 +08:00
OG T
4b24ecd67f fix(sprint3): 首席架構師 Review C1/C2/C3/M3/m1 修正 
C1: _ssh_execute 直接接收 key_path 參數，不反查 LAYER_SSH_CONFIG
C2: PlaybookService.create() proxy，Router 不再穿透呼叫 _repository
C3: CD Step 1b sed 替換 IMAGE_TAG_PLACEHOLDER，消除失敗中斷風險
M3: repair-bot 110/188 regex 統一 [a-z0-9][a-z0-9-]{0,30}，禁止底線
m1: defaultMode 0400 加八進位說明注釋
m2: _ssh_execute 用 deadline 計算剩餘 timeout

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 13:07:59 +08:00
OG T
4b934bb9fd feat(k8s): API Pod 掛載 repair SSH key (Task 13) 
- 06-deployment-api.yaml: volumeMount /etc/repair-ssh + volumes secret defaultMode 0400
- 對應 K8s Secret: awoooi-repair-ssh-key

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 11:47:37 +08:00
OG T
892c5d53a7 k8s(secret): 加入 repair SSH key 建立說明 template 
實際私鑰透過 kubectl create secret 手動建立，不上 Git
主機 110 (wooo) / 188 (ollama) 已設定 command= 受限 authorized_keys
SSH health check 驗證: REPAIR_BOT_HEALTHY:110 / REPAIR_BOT_HEALTHY:188

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 11:17:57 +08:00
OG T
ad4abefcd9 fix(k8s+ops): 修復告警鏈路 + Gitea runner 自動啟動
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 21s
Details 
## 修復項目

1. NetworkPolicy allow-nginx-ingress 加入 192.168.0.110
   - Alertmanager (在 110) 需要從 110 直接 POST webhook 到 API pod
   - 修復前: 110 被 NetworkPolicy default-deny 阻擋，webhook timeout
   - 修復後: 110 加入 ingress 白名單，告警鏈路恢復

2. awoooi-startup-110.sh 加入 Gitea Act Runner
   - Step 6: 啟動 /home/wooo/act-runner (gitea-runner container)
   - 修復前: 重開機後 runner 離線，CD pipeline 全面失效
   - 修復後: runner 自動重啟，若配置過期自動清除重新註冊

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 01:42:52 +08:00
OG T
3455044457 feat(phase25): Nemotron 主動防禦三方向 P0+P1+P2 完整實作
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 38s
Details
Type Sync Check / check-type-sync (push) Failing after 35s
Details 
P0 - DIAGNOSE Privacy-First Routing:
- ai_router.py: _local_fallback_chain [NEMOTRON→OLLAMA→REJECT]
- DIAGNOSE 意圖 override 改為 NEMOTRON (原 OLLAMA)
- DIAGNOSE fallback 使用 local-only 鏈，不觸碰雲端
- 全部失敗時 REJECT + Telegram 通知
- config.py: NEMOTRON_DIAGNOSE_TIMEOUT_SECONDS=30, OLLAMA_DIAGNOSE_TIMEOUT_SECONDS=60
- nemotron.py: 根據 context[task_type] 選擇 timeout

P1 - Knowledge Auto-Harvesting:
- models/knowledge.py: EntryType.AUTO_RUNBOOK + ANTI_PATTERN + symptoms_hash
- EntryStatus.PUBLISHED (ANTI_PATTERN 直接發布，無需審核)
- models/playbook.py: SymptomPattern.compute_hash() (16字元確定性 hash)
- services/runbook_generator.py: NemotronRunbookGenerator (v1.1)
  - generate_runbook() → AUTO_RUNBOOK (DRAFT) + Telegram 審核 card
  - generate_anti_pattern() → ANTI_PATTERN (PUBLISHED) + Telegram 通知
  - 使用 nvidia.chat() (正確介面)，Nemotron 超時時 Minimal fallback
- knowledge_service.py: check_anti_pattern(symptoms_hash, days=7)
- db/models.py: symptoms_hash VARCHAR(16) + ix_knowledge_symptoms_hash
- repositories/knowledge_repository.py: create() 支援 symptoms_hash + status
- auto_repair_service.py: anti_pattern_gate 在 decide() + runbook hook 在 execute()
- migrations/phase8_symptoms_hash.sql: ALTER TABLE + partial index + PUBLISHED constraint

P2 - Config Drift Detection:
- models/drift.py: DriftItem/DriftReport/DriftLevel/DriftIntent/DriftStatus
- services/drift_detector.py: GitStateReader + K8sStateReader + DriftDetector
- services/drift_analyzer.py: 白名單過濾 + DriftLevel 分級
- services/drift_interpreter.py: NemotronDriftInterpreter（意圖分析，不生成修復指令）
- services/drift_remediator.py: rollback(kubectl apply) + adopt(git push gitea)
- api/v1/drift.py: POST /scan, GET /reports, POST /rollback, POST /adopt
- migrations/phase9_drift_reports.sql: drift_reports 表
- k8s/drift-cronjob.yaml: 每小時自動掃描 CronJob

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-04 12:35:05 +08:00
OG T
c65ed5b1c9 feat(telegram): SRE 戰情室群組三頭政治 Triumvirate (ADR-053)
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 7m6s
Details 
- config.py: 新增 OPENCLAW_BOT_TOKEN / NEMOTRON_BOT_TOKEN / SRE_GROUP_CHAT_ID
- telegram_gateway.py: send_to_group / send_as_openclaw / send_as_nemotron / trigger_group_ai_discussion / _send_approval_card_to_group
- send_approval_card 告警發送後非同步觸發群組 AI 雙向討論
- configmap: SRE_GROUP_CHAT_ID=-1003711974679
- secrets: OPENCLAW_BOT_TOKEN / NEMOTRON_BOT_TOKEN CHANGE_ME 佔位

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-03 17:16:05 +08:00