wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
909 Commits 199 Branches 2 Tags
e82d3802c5f7df0e5eb092ff77062e3928441fdc
Commit Graph

88 Commits

Author SHA1 Message Date
OG T
af07c23675 fix(k8s): known_hosts 改掛 /etc/repair-known-hosts 獨立目錄，修 mount 衝突
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 12m11s
Details
E2E Health Check / e2e-health (push) Successful in 34s
Details 
/etc/repair-ssh 已被 repair-ssh-key 佔用，subPath 檔案掛載衝突
改為獨立目錄 /etc/repair-known-hosts，路徑同步更新 KNOWN_HOSTS_PATH

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-06 15:06:28 +08:00
OG T
d56aae135d fix(k8s): repair-known-hosts secret optional:true — Pod 不阻塞等待 secret 建立
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 8m35s
Details 
CD 首次跑時才建立 secret，optional 讓 Pod 先起來
等 CD 建立 secret 後自動掛載

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-06 14:48:45 +08:00
OG T
d4cb9a4ac5 ops(k8s): known_hosts Secret + Ansible 白名單 ConfigMap (Sprint 3 T2) 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-06 14:20:14 +08:00
OG T
23364423fa feat(webhook): ADR-059 GitHub → Gitea Webhook 遷移完成 
- gitea_webhook.py: Header 全部改 X-Gitea-*，移除 workflow_run handler
- gitea_webhook_service.py: _fetch_pr_diff 改直接 httpx，不依賴 github_api_service
- 清除兩個檔案的所有殘留 github_ log key，review_id prefix 改 gitea-
- test_gitea_webhook.py: 10/10 通過，docstring 修正
- 03-secrets.yaml: 新增 GITEA_WEBHOOK_SECRET 佔位
- cd.yaml: 新增 GITEA_WEBHOOK_SECRET 注入步驟
- ADR-059: 建立架構決策文件

待統帥操作: Gitea Actions secret + Gitea UI Webhook 設定

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 14:44:32 +08:00
OG T
4b24ecd67f fix(sprint3): 首席架構師 Review C1/C2/C3/M3/m1 修正 
C1: _ssh_execute 直接接收 key_path 參數，不反查 LAYER_SSH_CONFIG
C2: PlaybookService.create() proxy，Router 不再穿透呼叫 _repository
C3: CD Step 1b sed 替換 IMAGE_TAG_PLACEHOLDER，消除失敗中斷風險
M3: repair-bot 110/188 regex 統一 [a-z0-9][a-z0-9-]{0,30}，禁止底線
m1: defaultMode 0400 加八進位說明注釋
m2: _ssh_execute 用 deadline 計算剩餘 timeout

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 13:07:59 +08:00
OG T
4b934bb9fd feat(k8s): API Pod 掛載 repair SSH key (Task 13) 
- 06-deployment-api.yaml: volumeMount /etc/repair-ssh + volumes secret defaultMode 0400
- 對應 K8s Secret: awoooi-repair-ssh-key

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 11:47:37 +08:00
OG T
892c5d53a7 k8s(secret): 加入 repair SSH key 建立說明 template 
實際私鑰透過 kubectl create secret 手動建立，不上 Git
主機 110 (wooo) / 188 (ollama) 已設定 command= 受限 authorized_keys
SSH health check 驗證: REPAIR_BOT_HEALTHY:110 / REPAIR_BOT_HEALTHY:188

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 11:17:57 +08:00
OG T
ad4abefcd9 fix(k8s+ops): 修復告警鏈路 + Gitea runner 自動啟動
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 21s
Details 
## 修復項目

1. NetworkPolicy allow-nginx-ingress 加入 192.168.0.110
   - Alertmanager (在 110) 需要從 110 直接 POST webhook 到 API pod
   - 修復前: 110 被 NetworkPolicy default-deny 阻擋，webhook timeout
   - 修復後: 110 加入 ingress 白名單，告警鏈路恢復

2. awoooi-startup-110.sh 加入 Gitea Act Runner
   - Step 6: 啟動 /home/wooo/act-runner (gitea-runner container)
   - 修復前: 重開機後 runner 離線，CD pipeline 全面失效
   - 修復後: runner 自動重啟，若配置過期自動清除重新註冊

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 01:42:52 +08:00
OG T
3455044457 feat(phase25): Nemotron 主動防禦三方向 P0+P1+P2 完整實作
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 38s
Details
Type Sync Check / check-type-sync (push) Failing after 35s
Details 
P0 - DIAGNOSE Privacy-First Routing:
- ai_router.py: _local_fallback_chain [NEMOTRON→OLLAMA→REJECT]
- DIAGNOSE 意圖 override 改為 NEMOTRON (原 OLLAMA)
- DIAGNOSE fallback 使用 local-only 鏈，不觸碰雲端
- 全部失敗時 REJECT + Telegram 通知
- config.py: NEMOTRON_DIAGNOSE_TIMEOUT_SECONDS=30, OLLAMA_DIAGNOSE_TIMEOUT_SECONDS=60
- nemotron.py: 根據 context[task_type] 選擇 timeout

P1 - Knowledge Auto-Harvesting:
- models/knowledge.py: EntryType.AUTO_RUNBOOK + ANTI_PATTERN + symptoms_hash
- EntryStatus.PUBLISHED (ANTI_PATTERN 直接發布，無需審核)
- models/playbook.py: SymptomPattern.compute_hash() (16字元確定性 hash)
- services/runbook_generator.py: NemotronRunbookGenerator (v1.1)
  - generate_runbook() → AUTO_RUNBOOK (DRAFT) + Telegram 審核 card
  - generate_anti_pattern() → ANTI_PATTERN (PUBLISHED) + Telegram 通知
  - 使用 nvidia.chat() (正確介面)，Nemotron 超時時 Minimal fallback
- knowledge_service.py: check_anti_pattern(symptoms_hash, days=7)
- db/models.py: symptoms_hash VARCHAR(16) + ix_knowledge_symptoms_hash
- repositories/knowledge_repository.py: create() 支援 symptoms_hash + status
- auto_repair_service.py: anti_pattern_gate 在 decide() + runbook hook 在 execute()
- migrations/phase8_symptoms_hash.sql: ALTER TABLE + partial index + PUBLISHED constraint

P2 - Config Drift Detection:
- models/drift.py: DriftItem/DriftReport/DriftLevel/DriftIntent/DriftStatus
- services/drift_detector.py: GitStateReader + K8sStateReader + DriftDetector
- services/drift_analyzer.py: 白名單過濾 + DriftLevel 分級
- services/drift_interpreter.py: NemotronDriftInterpreter（意圖分析，不生成修復指令）
- services/drift_remediator.py: rollback(kubectl apply) + adopt(git push gitea)
- api/v1/drift.py: POST /scan, GET /reports, POST /rollback, POST /adopt
- migrations/phase9_drift_reports.sql: drift_reports 表
- k8s/drift-cronjob.yaml: 每小時自動掃描 CronJob

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-04 12:35:05 +08:00
OG T
c65ed5b1c9 feat(telegram): SRE 戰情室群組三頭政治 Triumvirate (ADR-053)
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 7m6s
Details 
- config.py: 新增 OPENCLAW_BOT_TOKEN / NEMOTRON_BOT_TOKEN / SRE_GROUP_CHAT_ID
- telegram_gateway.py: send_to_group / send_as_openclaw / send_as_nemotron / trigger_group_ai_discussion / _send_approval_card_to_group
- send_approval_card 告警發送後非同步觸發群組 AI 雙向討論
- configmap: SRE_GROUP_CHAT_ID=-1003711974679
- secrets: OPENCLAW_BOT_TOKEN / NEMOTRON_BOT_TOKEN CHANGE_ME 佔位

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-03 17:16:05 +08:00
OG T
ff5a77f7a9 fix(telegram): 啟用 Polling + 修正 InfraAlertMessage 格式
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 6m52s
Details 
1. TELEGRAM_ENABLE_POLLING: false→true
   - clawbot-v5 已停止 polling (STANDBY_MODE)
   - AWOOOI API 接管，統帥可與 OpenClaw/NemoClaw 雙 AI 對話

2. InfraAlertMessage.format() 加入 note 欄位
   - NIM 慢屬正常不再顯示「自動修復失敗」
   - 改為 💡 資訊性提示

3. NIM 探測端點改為 /v1/models (輕量，不觸發計費)
   timeout: 10s → 25s (NIM 免費 tier 冷啟動)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-03 16:43:40 +08:00
OG T
4284337249 fix(config): NEMOTRON_TIMEOUT_SECONDS 30→55 固化到 ConfigMap
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 7m0s
Details 
NIM 免費 tier 延遲 11-45s，30s 硬編碼導致所有慢請求超時。
已同步 prod/dev ConfigMap，避免下次 CD 部署被覆蓋。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-03 15:58:11 +08:00
OG T
48c65756da chore(config): USE_AI_ROUTER=true 寫入 ConfigMap (Phase 24 B2)
Some checks failed
E2E Health Check / e2e-health (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
防止下次 CD deploy 覆蓋 kubectl set env 的設定。
B2 觀察期 48h, 截止 2026-04-04 18:40 台北時間。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-02 21:26:53 +08:00
OG T
3f339110dd fix(observability): 同步 .188 實際部署調整至 repo
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
E2E Health Check / e2e-health (push) Has been cancelled
Details 
與原始計畫的差異:

1. MinIO Bearer Token 認證
   - 原計畫: MINIO_PROMETHEUS_AUTH_TYPE=public (此版本不支援)
   - 實際: mc admin prometheus generate 產生 Bearer Token
   - 更新: prometheus-config-phase-o.yaml 加入 bearer_token

2. remote_write 廢棄 → OTEL Collector Prometheus scrape
   - 原計畫: Prometheus remote_write → SigNoz OTEL /api/v1/write
   - 實際: SigNoz OTEL Collector 不支援 Prometheus remote_write 格式 (404)
   - 改用: OTEL Collector prometheus receiver 直接 scrape node-exporter + kube-state-metrics
   - 新增: ops/signoz/otel-collector-config-phase-o.yaml (版本控管副本)

3. ADR-053 驗收清單更新為實際結果

Co-Authored-By: Claude Code <noreply@anthropic.com>
 2026-04-02 21:23:47 +08:00
OG T
99be215e83 fix(monitoring): R1 Review 修正 — Blackbox DNS/PSA label/告警閾值 
Critical: Blackbox Exporter replacement 從 K8s DNS 改為主機 IP (192.168.0.188:9115)
Important: Descheduler namespace 顯式宣告 PSA restricted labels
Suggestion: failedJobsHistoryLimit 3→1, 新增 MinioDiskUsageCritical 5% 告警

R1 Review by: 首席架構師 (Phase O-1)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 14:02:50 +08:00
OG T
41bf0681cf feat(observability): Phase O-2/O-3 OTEL Log管線 + Event Exporter + Remote Write 
O-2.1: OTEL Collector DaemonSet (filelog receiver)
  - 收集所有 K3s 節點 Pod stdout/stderr → SigNoz ClickHouse
  - CRI log parser (Go time layout for +08:00 timezone)
  - filter processor 排除 kube-system debug noise
  - observability namespace PSA privileged (log 目錄需 root)
  - 資源限制: 50m-200m CPU / 64-128Mi Memory

O-2.2: kubernetes-event-exporter
  - K8s Event → 結構化 JSON Log → SigNoz
  - Warning/Error 全量保留, Normal 過濾高頻事件
  - 解決: Event 預設僅保留 ~1hr 的致命盲區

O-3: Prometheus remote_write 配置模板
  - 白名單: ~50 關鍵 metric series (node/container/kube/api/db)
  - 目標: 90 天長期儲存於 SigNoz ClickHouse

已部署驗證: 3 Pod Running, 0 error, filelog 正常監控所有 namespace

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 14:01:42 +08:00
OG T
6ce82ff883 fix(k3s): Phase O-1 基礎設施修復 — Descheduler + MinIO/Kali 監控 
O-1.1: Descheduler securityContext 修復 (PodSecurity restricted 合規)
  - 新增 pod securityContext (runAsNonRoot, runAsUser:65534, seccompProfile)
  - 新增 container securityContext (allowPrivilegeEscalation:false, drop ALL)
  - 補齊 RBAC: namespaces + replicasets list 權限
  - 已部署驗證: CronJob 成功執行 (Status: Completed)

O-1.3: MinIO Prometheus scrape 配置 + 告警規則
O-1.4: Kali Blackbox TCP probe + 告警規則
  - MinioDown, MinioDiskUsageHigh, MinioOfflineDisk
  - KaliScannerDown

待手動部署: Prometheus config → .188, kubectl kubeconfig → 120/121

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 13:55:26 +08:00
OG T
73e8f8ab77 feat(ai): Phase 24-A+B1 — AI Provider Registry + 絞殺者包裝 (ADR-052)
Some checks failed
E2E Health Check / e2e-health (push) Successful in 16s
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
Brain Layer 雙軌 Registry 架構:
- 新建 src/services/ai_providers/ 目錄 (interfaces + 4 providers)
  - OllamaProvider (local, rca/chat/code_review)
  - GeminiProvider (cloud, rca/chat)
  - ClaudeProvider (cloud, rca/chat/code_review)
  - OpenClawNemoProvider (cloud, rca — 委派 188→NIM)
- 擴展 ai_router.py 加入:
  - AIProviderRegistry (動態註冊/啟停)
  - AIRouterExecutor (Cache + 閘門 CB/RL/Sem + 執行)
- openclaw.py 絞殺者包裝: USE_AI_ROUTER=true 走新路徑
- config.py + ConfigMap 加入 USE_AI_ROUTER=false (安全預設)
- ADR-052 正式文件 (14 項決策 D1-D14)
- HARD_RULES v1.7 加入 AI Router 規範

安全: USE_AI_ROUTER=false 預設不啟用，需手動開啟觀察
回滾: kubectl set env deployment/awoooi-api USE_AI_ROUTER=false

2026-04-02 ogt: Phase 24 首批實作

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 13:16:09 +08:00
OG T
3ecfe7b3f5 chore: 清理 NemoNodeAnimation 殘留 + 修復 Migration YAML
Some checks failed
E2E Health Check / e2e-health (push) Successful in 19s
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
- 移除 nemo-node-animation.tsx (無人引用，已被 NemoClaw 取代)
- Migration YAML: 修復 $$ 在 YAML heredoc 被 shell 解析問題
  改用單引號字串 DO '' 語法

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 09:09:25 +08:00
OG T
d8be78b135 feat(api): Knowledge Base Phase 1 後端四層架構
Some checks failed
CD Pipeline / build-and-deploy (push) Successful in 7m0s
Details
E2E Health Check / e2e-health (push) Successful in 17s
Details
Type Sync Check / check-type-sync (push) Failing after 30s
Details 
- models/knowledge.py: Pydantic Schema (EntryType/Source/Status/CRUD)
- db/models.py: KnowledgeEntryRecord ORM (PostgreSQL)
- repositories/interfaces.py: IKnowledgeRepository Protocol
- repositories/knowledge_repository.py: PostgreSQL CRUD 實作
- services/knowledge_service.py: 業務邏輯 (get_db_context 內部管理 session)
- api/v1/knowledge.py: REST Router (get_knowledge_service，無直接 DB 存取)
- main.py: 掛載 Knowledge Base Router
- k8s/jobs/migrate-knowledge-entries.yaml: DB Migration Job

API 端點: GET/POST / | GET/PATCH/DELETE /{id} | POST /{id}/approve
         GET /search | GET /categories

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-02 00:55:56 +08:00
OG T
9cf73bda4f feat(llmops): 啟用 Langfuse LLMOps 追蹤 + CD 自動注入 Keys
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 7m6s
Details
E2E Health Check / e2e-health (push) Successful in 18s
Details 
- 04-configmap.yaml: LANGFUSE_ENABLED=true (Phase 15.1 Key 已在 K8s Secret)
- cd.yaml: 補齊 Langfuse keys CD 自動注入 (LANGFUSE_PUBLIC/SECRET_KEY)
- LOGBOOK.md: ClawBot → OpenClaw 命名修正
- .gitignore: 加入 tsconfig.tsbuildinfo + .superpowers/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 22:19:22 +08:00
OG T
dc9a07bf20 fix(k8s): NetworkPolicy 修正 OpenClaw port 8089→8088 (clawbot-v5 使用 8088)
Some checks failed
E2E Health Check / e2e-health (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
2026-04-01 ogt: OpenClaw 從 openclaw container (8089) 遷移到 clawbot (8088)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 22:17:09 +08:00
OG T
5809d3e336 feat(ai): 委派 Incident RCA 給 OpenClaw (Nemo) — 架構鐵律修正
Some checks failed
E2E Health Check / e2e-health (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
架構鐵律: OpenClaw = AI 大腦，AWOOOI API 透過 HTTP 委派仲裁
修改:
- openclaw.py: 加入 _call_openclaw_analyze()，在 LLM fallback 前先呼叫 OpenClaw
- 04-configmap.yaml: OPENCLAW_URL 修正為 :8088 (新容器 port)
- AI_FALLBACK_ORDER 改為 ["ollama","claude"] (移除 Gemini 付費 API)

OpenClaw /api/v1/analyze/incident → qwen2.5:7b 本機 Ollama (Nemo)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 21:11:30 +08:00
OG T
bfee94bb6f fix(ai): ADR-044 修正 AI_FALLBACK_ORDER — Gemini 優先做 RCA 仲裁
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 6m49s
Details
E2E Health Check / e2e-health (push) Successful in 21s
Details 
問題連鎖:
1. NVIDIA nemotron-mini-4b JSON 截斷 → proposal_parse_failed
2. Ollama K8s 並發請求 → GPU 排隊逾時 (90s)
3. Expert System fallback → 信心 0%

ADR-044: OpenClaw RCA 應用 Ollama/Gemini，非 NVIDIA Tool Calling 模型
修正: gemini > ollama > nvidia > claude

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 20:35:42 +08:00
OG T
555e808f39 fix(ai): ollama 優先於 nvidia — 修復 nemotron-mini-4b JSON 截斷導致 0% 信心
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 6m42s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details 
根本原因: nemotron-mini-4b-instruct 輸出 JSON 被截斷 (raw_response={"confidence": )
→ proposal_parse_failed → fallback Expert System → AI 仲裁 0% 信心

修復: AI_FALLBACK_ORDER 改為 ollama 優先，NVIDIA 降為第二
(Ollama qwen2.5:7b-instruct 在 192.168.0.188:11434 輸出品質穩定)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 19:22:15 +08:00
OG T
5b938887c0 fix(telegram): 關閉 K8s prod TELEGRAM_ENABLE_POLLING 解決 409 Conflict
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 6m48s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details 
AWOOOI API 與 OpenClaw(192.168.0.188) 同時 Long Polling 造成 409 Conflict，
導致 AI 仲裁降級為規則匹配(0%信心)。

架構原則: OpenClaw 是唯一 Telegram Gateway，K8s 只發送訊息。

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 18:09:47 +08:00
OG T
71a4e0f8c8 fix(k8s): 修復 dev RBAC RoleBinding 欄位名稱錯誤
Some checks failed
CD Pipeline / build-and-deploy (push) Successful in 6m54s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details
CD Pipeline (Dev) / build-and-deploy-dev (push) Failing after 3m53s
Details 
apiRef → name (正確 Kubernetes 欄位名稱)
防止 RoleBinding 建立失敗

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 16:27:12 +08:00
OG T
9913f5dc6d feat(infra): 開發環境分離 + BuildKit cache 修復 + circuit breaker 優化
Some checks failed
CD Pipeline / build-and-deploy (push) Successful in 6m52s
Details
E2E Health Check / e2e-health (push) Successful in 17s
Details
CD Pipeline (Dev) / build-and-deploy-dev (push) Failing after 9s
Details 
1. k8s/awoooi-dev/: 新建 dev namespace (01-05 配置)
   - Namespace + ResourceQuota (cpu 2/4, mem 4Gi/8Gi)
   - ConfigMap: ENVIRONMENT=dev, LOG_LEVEL=DEBUG, SHADOW_MODE=false
   - Deployment: 1 replica, NodePort 32344, image dev-latest
   - RBAC: awoooi-executor-dev ServiceAccount

2. .gitea/workflows/cd-dev.yaml: dev branch CD pipeline
   - 觸發: dev branch push
   - Build: --no-cache (防 cache poisoning)
   - Tag: dev-{sha} / dev-latest
   - Deploy: awoooi-dev namespace, health check 32344
   - Telegram: [DEV] 前綴通知

3. apps/api/Dockerfile: ARG CACHE_BUST=none (防 BuildKit cache 毒化)
   - deps 層 (pip install) 仍可 cache
   - src/ 和 models.json 層每次重建

4. .gitea/workflows/cd.yaml: 正式環境 API build 加入 CACHE_BUST=git_sha
   - 確保 models.json 等配置變更正確進入 image

5. apps/api/src/services/nvidia_provider.py: timeout 不計入 circuit breaker
   - TimeoutException → 只 log，不 record_failure()
   - 只有硬性錯誤 (auth/rate limit/exception) 才斷路

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 16:22:21 +08:00
OG T
419dc2f8e0 fix(nvidia): timeout 60s→30s，NVIDIA 第一保免費，失敗轉 Gemini
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 5m46s
Details
E2E Health Check / e2e-health (push) Successful in 16s
Details 
- nvidia_provider.py: NVIDIA_TIMEOUT 60→30s
- models.json: timeout_seconds 60→30s
- configmap: NEMOTRON_TIMEOUT_SECONDS 45→30s, fallback 恢復 nvidia 第一
目標: Nemo 有足夠時間回應(free)，失敗快速轉 Gemini(備援)，整體機制可運作

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 16:05:19 +08:00
OG T
4c622813af fix(auto-repair): 實際可用的自動修復門檻 (Phase 22 P1)
Some checks failed
E2E Health Check / e2e-health (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details 
問題: 四道鎖全卡死導致自動修復永遠不觸發
1. configmap: Gemini 排第一 (100ms vs NVIDIA 60s timeout)
2. auto_approve: confidence 0.90→0.65, trust 5→1, playbook 3→1
3. auto_approve: 開放 medium 風險, require_playbook=False

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 16:02:16 +08:00
OG T
eccf61fbc9 fix(ai): 修復假信心度 + 解除 Shadow Mode (Phase 22 P1)
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
E2E Health Check / e2e-health (push) Has been cancelled
Details 
1. openclaw.py: LLM 截斷時 confidence 0.82→0.0 (禁止偽造信心度)
2. prompts.py: NEMOTRON schema 範例值改用佔位符，防模型照抄 0.75
3. configmap: SHADOW_MODE_ENABLED=false，開放 low 風險自動執行
   條件門檻: confidence≥90% + trust_score≥5 + playbook_success≥95%

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-01 15:59:42 +08:00
OG T
cc6b18e3bc fix(phase22): 修復 Telegram 對話三個 Bug (ADR-044)
All checks were successful
E2E Health Check / e2e-health (push) Successful in 18s
Details 
P0: security_interceptor.py 新增 intercept_telegram() 方法
- 修復 _handle_chat_message 的 AttributeError (致命 Bug)
- 白名單驗證，不需要 Nonce (對話訊息 vs 按鈕回調)

P1: nvidia_provider.py chat() 新增 use_json_mode 參數
- 對話場景預設 False (自然語言回應)
- RCA/分析場景傳入 True (結構化 JSON 輸出)
- openclaw.py RCA 呼叫加上 use_json_mode=True

P2: K8s ConfigMap 啟用 TELEGRAM_ENABLE_POLLING=true
- K8s AWOOOI API 接管 @tsenyangbot Long Polling
- OpenClaw (188) 停止 Telegram，改為純 REST 服務

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-31 21:53:09 +08:00
OG T
6da240219d feat(k8s): Phase 22 啟用 OpenClaw + Nemotron 協作 (ADR-044)
All checks were successful
E2E Health Check / e2e-health (push) Successful in 17s
Details 
- 新增 ENABLE_NEMOTRON_COLLABORATION=true
- 新增 NEMOTRON_TIMEOUT_SECONDS=45
- 新增 NEMOTRON_ASYNC_UPDATE=true
- 統帥需求: @tsenyangbot 同時顯示 OpenClaw 仲裁 + Nemotron 執行

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-31 21:39:14 +08:00
OG T
219525f64f fix(k8s): #33 Sentry + OTEL 修復
All checks were successful
E2E Health Check / e2e-health (push) Successful in 17s
Details 
修復項目:
1. SENTRY_DSN: K8s secret 已修補 (空值 → 正確 DSN)
2. OTEL_EXPORTER_OTLP_ENDPOINT: 24318 → 24317 (gRPC 端口)

驗證結果:
- sentry_initialized ✅
- OTEL export 無錯誤 ✅
- Session Replay 配置 ✅

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-31 12:18:06 +08:00
OG T
723e8ef251 feat(api): Phase 21.3 Weekly Report (ADR-041)
All checks were successful
E2E Health Check / e2e-health (push) Successful in 16s
Details 
- 新增 WeeklyReportMessage dataclass (telegram_gateway.py)
- 新增 WeeklyReportService (整合 StatsService + K3sMonitor)
- 新增 CronJob (每週五 18:00 台北)
- 新增 API 端點 (/stats/weekly/preview, /stats/weekly/report)

Phase 21 定期報告機制全部完成！
- 21.1 Daily E2E Schedule ✅
- 21.2 K3s Telegram Report ✅
- 21.3 Weekly Report ✅

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-31 11:28:46 +08:00
OG T
ce6b1b1c64 docs: 更新 LOGBOOK - #17 i18n Hydration 完成 
前端 P1 改進全部完成:
- #15 SSE + 樂觀更新 (8c8664c)
- #16 DOM Bypass (0b87018)
- #17 i18n Hydration (f25e94e)

首席架構師審查: 96/100 OUTSTANDING

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-31 11:23:38 +08:00
OG T
fbc670036d fix(ai): NVIDIA Nemotron 優先仲裁 
2026-03-30 ogt: AI Fallback 順序修正
- ["nvidia","gemini","ollama","claude"]
- Nemotron Tool Calling 83.3% 精準度
- 修復 Gemini 仍為首位問題

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-30 01:12:28 +08:00
OG T
89f0bae3f2 feat(safety-net): complete wave 1 atomicity (adr-038, adr-039, debounce, graceful degrade, xclaim)
Some checks failed
E2E Health Check / e2e-health (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
2026-03-29 23:55:38 +08:00
OG T
79134fb019 feat(ai): 新增 NVIDIA Nemotron 到告警 Fallback Chain 
- 新增 _call_nvidia() 一般告警支援 (非 Tool Calling)
- Fallback 順序: Gemini → Nvidia → Ollama → Claude
- Nvidia 免費 tier ($0)，含 Token 追蹤

解決: Gemini 超限 (500/500) 後無法 fallback 問題

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 20:28:24 +08:00
OG T
b97f9364fb feat(k8s): add Worker HPA + fix non-AI confidence values 
Wave 2 Deployment:
- Worker HPA: min:1 max:3, CPU 70%, Memory 80%
- 前置條件: XCLAIM + terminationGracePeriodSeconds:90 (Wave 1 ✅)
- 比 API/Web 更保守的擴縮策略 (120s up, 600s down)

Confidence Fix:
- 非 AI 分析來源 (fallback/playbook/historical/consensus) 設 confidence=0.0
- 避免混淆 AI 信心度與其他指標 (成功率/相似度)
- 涉及: github_webhook, decision_manager, intent_classifier, learning_service

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 16:09:37 +08:00
OG T
a5a6bd3408 feat(monitoring): K8s alert rules + Grafana dashboards + ops 腳本 
- k8s/monitoring/alert-chain-monitor.yaml
- k8s/monitoring/database-alerts.yaml
- ops/grafana/ Grafana dashboards
- ops/signoz/ SignOz 配置
- ops/scripts/ 維運腳本

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 16:04:14 +08:00
OG T
237fb64a81 chore(k8s): secrets template + web deployment 更新 
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 16:03:47 +08:00
OG T
39396dc57a feat(worker): Wave 1 Signal Worker XCLAIM + Graceful Shutdown 
ADR-038/039 Wave 1 強化:
- 新增 Active Sweeper: XPENDING + XCLAIM 回收閒置訊息
- PENDING_IDLE_MS: 60秒無ACK則可被回收
- SWEEP_INTERVAL_S: 每30秒掃描一次
- Graceful Shutdown: 75秒超時 (搭配 K8s 90秒)
- 超過 MAX_RETRIES 的訊息強制 ACK

K8s Worker Deployment:
- 新增 terminationGracePeriodSeconds: 90

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:53:05 +08:00
OG T
12e49d844a feat(monitoring): ADR-037 Wave B - Database Exporters + Prometheus 整合 
- 部署 PostgreSQL Exporter (192.168.0.188:9187)
- 部署 Redis Exporter (192.168.0.188:9121)
- 更新 Prometheus scrape config
- 首席架構師審查: 97% OUTSTANDING

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 15:18:54 +08:00
OG T
93c3280481 feat(monitoring): Phase 20 Nemotron 完整監控整合 
服務註冊表:
- 新增 nvidia-nemotron AI 服務
- 3 個 Prometheus metrics 定義
- 4 個告警規則 (circuit_breaker, timeout, error_rate, rate_limit)
- fallback 策略 (nvidia → gemini → ollama)

Alertmanager 規則:
- NvidiaCircuitBreakerOpen (P1)
- NvidiaToolCallingHighLatency (P2)
- NvidiaToolCallingHighErrorRate (P0)
- NvidiaCircuitBreakerHalfOpen (Info)
- NvidiaCircuitBreakerClosed (Info)
- NvidiaNoRequests (P3)

自動修復:
- fallback_to_gemini
- fallback_to_ollama
- switch_model

ADR: ADR-036

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 02:05:59 +08:00
OG T
179e659f14 chore: 清理 Playwright 產物 + kube-state-metrics 告警擴充 
清理工作:
- .gitignore 新增 playwright-report/ 和 test-results/ 排除
- 保留 phase19/ 參考截圖目錄

kube-state-metrics 告警擴充 (P3):
- CronJobLastRunFailed: Job 執行失敗
- DaemonSetMissingPods: DaemonSet 缺少 Pod
- StatefulSetReplicasMismatch: StatefulSet 副本不足
- ContainerWaiting: ImagePullBackOff/CrashLoopBackOff 偵測
- PDBViolation: PDB 健康 Pod 數不足
- NodeUnschedulable: 節點標記為不可排程

新增:
- apps/api/scripts/test_nemotron_tool_calling.py (E2E 比較測試)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 01:28:35 +08:00
OG T
725392b578 fix(k8s): NetworkPolicy 繞過 kustomize commonLabels 
問題: kustomize commonLabels 會加到 NetworkPolicy egress[].to[].podSelector
      導致 DNS rule 要求 CoreDNS pods 有 system:awoooi + environment:prod
      但 CoreDNS 只有 k8s-app:kube-dns，造成 DNS 解析失敗

修復:
- kustomization.yaml: 移除 02-network-policy.yaml
- cd.yaml: 新增 Apply NetworkPolicy step 單獨套用

2026-03-29 ogt: 根本原因修復

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 01:27:29 +08:00
OG T
ee2bceefff feat(monitoring): Phase 19.6 測試文檔 + P1-P3 改進 + 首席架構師審查 
Phase 19.6 測試文檔收尾:
- E2E 測試擴充至 18 項 (Terminal/GenUI 驗證)
- 新增 PHASE19-VERIFICATION-CHECKLIST.md (完整驗證清單)

P1 驗證:
- ArgoCD Metrics NodePort 監控 (30883/30884)
- TLS 證書監控 (Blackbox Exporter 9115)

P2 改進:
- waitForTimeout → waitForLoadState('networkidle')
- 跨平台快捷鍵 (Meta+J / Control+J)
- SKIP_MULTISIG_TESTS 環境變數控制
- Prometheus GitOps 部署腳本

P3 改進:
- HPA maxReplicas 4 → 6 (API/Web)

首席架構師審查: 47/50 OUTSTANDING (94%)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-29 01:19:26 +08:00
OG T
dc7daf5d81 docs(monitoring): 更新 ArgoCD Metrics 端點文檔 
- ArgoCD Server Pod 運行在 mon1 (192.168.0.121)
- 更新 Prometheus target 為 192.168.0.121:30883
- 標記配置已部署並驗證

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 23:59:46 +08:00
OG T
e75e578547 feat(monitoring): P1/P2 改進 - ArgoCD Metrics + TLS 證書告警 
## P1: ArgoCD Metrics
- 新增 ArgoCD Metrics NodePort (30882, 30883)
- 更新 NetworkPolicy 允許 Prometheus (188) 抓取
- 提供 Prometheus scrape config 範本

## P1: NetworkPolicy AI API
- 文檔標註 K8s NetworkPolicy 不支援 FQDN 限制
- 維持現有配置避免 AI 功能中斷

## P2: TLS 證書告警
- 新增 TLSCertExpiringIn30Days (30天預警)
- 新增 TLSCertExpiringIn7Days (7天緊急)
- 新增 TLSCertExpired (已過期)
- 新增 TLSProbeFailure (探測失敗)

## P2: Multi-Sig E2E 測試
- 標記為條件式執行 (API 不可用時自動跳過)
- 避免 CI/CD 因無法連接生產 API 而失敗

首席架構師審查: 2026-03-29 (台北時間)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
 2026-03-28 23:48:57 +08:00