Your Name
6878e62af7
feat(flywheel): W1 PR-P1 + ADR-091 T1 — 飛輪 80→90 第一波
...
依 onboarder 端到端閉環審計挖出的 10 條斷鏈 + critic 鐵律違反全景,
W1 第一波修復飛輪鐵證 1 + 2 的核心斷鏈 C1。
## W1 PR-P1 — matched_playbook_id 四斷點守門 (C1 修復)
fullstack 探勘發現 4 斷點之前 session 已修,本 PR 補:
- ENABLE_PLAYBOOK_MATCHING feature flag (default=true)
rollback: kubectl set env deployment/awoooi-api ENABLE_PLAYBOOK_MATCHING=false
- proposal_service._try_playbook_match_id 入口加 flag check
- 7 個 e2e 測試補上保護網(之前無測試覆蓋)
斷鏈 C1 證據鏈:proposal_service.generate_proposal() → matched_playbook_id
→ approval_db → approval_repository → learning_service._update_playbook_stats
24h 後 playbooks.trust_score 應有真實 EWMA 更新。
## ADR-091 T1 — auto_generate_rule 雙寫 DB (鐵證 1 第一步)
飛輪鐵證 1:alert_rule_catalog.source='ai_generated' 全 codebase 0 筆。
auto_generate_rule() 寫 alert_rules.yaml 但不寫 DB → AI 自學成果與 catalog 雙軌脫鉤。
修法(依 ADR-091 §1 D1):
- 新增 _insert_catalog_ai_generated():YAML 寫入成功後雙寫
source='ai_generated', confidence=0.5, review_status='draft', created_by_agent
- 新增 _parse_for_to_seconds() helper("30s"/"5m"/"2h" → seconds)
- ON CONFLICT (rule_name) DO NOTHING 冪等保證
- transaction 策略:YAML + DB 不在同一 transaction(YAML 已成 SoT,DB 失敗只 log)
- ENABLE_AI_RULE_CATALOG_WRITE feature flag (default=true)
rollback: kubectl set env deployment/awoooi-api ENABLE_AI_RULE_CATALOG_WRITE=false
13 個測試覆蓋:parse helper 8 + 業務邏輯 5(success/db_fail/idempotent/flag/SQL_lit)
## 驗證
1572 unit tests 全綠(+20 新增:PR-P1 7 + ADR-091 T1 13)
## 期望影響
飛輪自主化評分:42 → 65(+23 = C1 +3 + 鐵證 1 +20)
## 已知債(critic PR review 揭示,下一個 commit 處理)
- KMWriter 統一契約 3 條 caller 路徑被旁路(C1/M1/M2)
- KMWriter 冪等聲明與實作不符(M3 缺 ON CONFLICT)
- Alertmanager equal:[] 爆炸抑制 + 版本未驗(M4/M5)
- drift checker regex 脆弱(M7 應改 AST)
- governance health score skipped 失真(M6)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-29 10:44:39 +08:00
Your Name
dc18b0ebd6
fix(prometheus_url): drift 殘存追修 — kured 守門員 + monitoring API
...
debugger 全 codebase 追根溯源後揪出 5 處 PROMETHEUS_URL drift 殘存
(根因:docs/reference/SERVICE-ENDPOINTS.md 早期把 Prometheus 標在 188
是整個 codebase drift 的源頭)。
本次修最急的 2 處:
## 🔴 🔴 kured.yaml:132(守門員失效風險)
- 188 → 110
- kured 跑 reboot 前會查 Prometheus alerts,連錯主機 = 跳過保護直接 reboot 主機
- 對齊 ConfigMap + config.py PROMETHEUS_URL
## 🟡 monitoring.py:67(單一事實源)
- 寫死 110:9090 改用 settings.PROMETHEUS_URL
- 主機巧合正確但繞過 ConfigMap 注入機制
- 未來 Prometheus 再遷移避免再次 drift
## 暫不修
- k3s_monitor_service.py:38 用 121:30090 是 K3s NodePort 內網端點
與外部 PROMETHEUS_URL 概念不同,需新增 PROMETHEUS_INTERNAL_URL setting
- 其他 docstring + 文件 drift(SERVICE-ENDPOINTS.md 等)留待後續
## 驗證
1552 unit tests 全綠(無回歸)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-29 10:44:39 +08:00
Your Name
6eb33594c2
docs(logbook): T0 12-Agent 全景驗證紀錄
...
承接前段 session wave2 (commit 143c15f0 ) + DB cleanup + Gitea HMAC + ArgoCD/Sentry MCP,
派四位專家並行驗證(critic / db-expert / debugger / tool-expert)。
詳情:B1/B2 鬼魂按鈕 + KM 早期吞例外 + M1-M4 中度問題 + G1-G3 環境治理 gap。
此 commit 主要為 LOGBOOK 索引補齊,本次 P0/P1 修復內容詳見前 2 個 commit。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-29 10:44:39 +08:00
Your Name
c22e5f334e
feat(km): P1-1 KMWriter 統一契約 + 5 caller 切換 + M4 反查鏈補齊
...
12-Agent 全景診斷揪出 KM 寫入鏈路 5 條入口無統一契約,fire-and-forget
在 Pod recycle 時會丟失條目。本次抽 KMWriter 強制 7 條契約。
## 7 條契約強制
1. 同步底線:強制 await asyncio.wait_for(timeout)
2. 重試:3 次指數退避 1s/2s/4s(OperationalError / 網路類例外)
3. 失敗回收:3 次後寫 Redis DLQ km:dlq + log
4. 觀測:structlog event + 預留 metric hook(P1-3 補 emitter)
5. 冪等:incident_id + path_type 為 unique key
6. 禁止吞例外:except 必須 log + raise/DLQ
7. M4 反查鏈:payload 含 approval_id 時自動填 related_approval_id 並回填 Path A
## Caller 切換(5 條入口統一介面)
- incident_service.py:1086 Path A(KB extractor + km_conversion)
- approval_execution.py:771 Path B-人工
- decision_manager.py:2178 Path B-自動成功(消除跨類私有方法調用 M1)
- decision_manager.py:2200 Path B-自動失敗(修 B2 早期吞例外)
- playbook_service.py:210 PlaybookKM(兩份 T0 報告都漏的第三條)
## M4 反查鏈補齊
- knowledge.py + models.py: 補 related_approval_id ORM 欄位
- 對齊 phase26_incident_km_integration.sql:20 schema(partial index 已存在)
- approval↔KM 雙向反查鏈完整(dual-path 縫合線)
## Feature Flag (rollback 保險)
- KM_WRITE_AWAIT=true (default): await + timeout + DLQ 強制
- KM_WRITE_AWAIT=false: fire-and-forget(舊行為)
## 測試
- apps/api/tests/test_km_writer.py: 18 測試全綠
覆蓋 success / timeout / retry / DLQ / 冪等 / KMWriteError /
on_failure=raise / 反查鏈回填
- 1552 unit tests 全綠(無回歸)
## 驗收
飛輪閉環核心 — KM 寫入不再靜默丟失,AI 學習鏈不斷裂。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-29 10:44:39 +08:00
Your Name
715dc3cb91
fix(observability): P0 假警報止血 + ConfigMap drift 對齊 + 治理工具
...
12-Agent 全景診斷觸發的 P0/P1 觀測層修復。
## P0 假警報止血(4 SLO 雪崩根因)
- governance_agent.py:306 — 空 result 不再 fallback 0.0,改 continue + log warning
根因:Prometheus 查無資料(emitter 未實作 / rule 未部署)被誤判為 SLO=0
必觸發 violated=True 噴 4 條假告警
## P0 鬼魂按鈕守門
- telegram_gateway.py:1654 — LLM 動態按鈕 Redis 失敗時 btn_list.clear()
first_row(批准/拒絕,HMAC nonce 無狀態)由 caller 1488 永遠保留
feedback_no_ghost_buttons.md 三缺一鐵律對齊
## ConfigMap drift 修復(3 處)
- config.py:683 PROMETHEUS_URL: 188→110(drift checker 揪出 = SPF-4 部分根因)
- config.py:705 ARGOCD_URL: 125→121(T0 G3 已知)
- config.py:375 AI_FALLBACK_ORDER: 補 nvidia 對齊 ConfigMap
## P1 Alertmanager 升級(amtool SUCCESS)
- ops/alertmanager/alertmanager.yml: deprecated → v0.27+ 新語法
- match/match_re → matchers
- source_match/target_match → source_matchers/target_matchers
- group_by 加 team label(防 SLO 雪崩 4 條同秒推)
- PostgreSQL/Redis inhibit 補 equal: ['instance'](防爆炸抑制)
- 新增 3 組因果抑制:
- OllamaInstanceDown → SLO_*/AI_*(30 分鐘)
- KMConverterDown → SLO_KMGrowthRate*
- SLO_*_FastBurn → SLO_*_(Medium|Slow)Burn
## 治理工具落地
- scripts/check_config_drift.py: ConfigMap vs code default drift 檢測
揪出 PROMETHEUS_URL drift 是 SPF-4 根因(governance_agent 連 188 而非 110)
- scripts/health_check_session.sh: 11 服務 + 4 SSH + drift + git 全景驗證
## 驗證
- 1552 unit tests 全綠
- amtool check-config SUCCESS(8 inhibit_rules / 2 receivers)
- drift checker 4 欄位全對齊
- health check 11 服務全可達
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-29 10:44:39 +08:00
AWOOOI CD
20009cddcf
chore(cd): deploy 143c15f [skip ci]
2026-04-28 07:36:19 +00:00
Your Name
143c15f052
feat(wave2+km): LLM 動態按鈕啟用 + KM 自動寫入 + AI Router dead code 標記
...
CD Pipeline / build-and-deploy (push) Successful in 9m52s
- ConfigMap: USE_LLM_DYNAMIC_BUTTONS=true(B2/B3/B4 handler 全就緒)
- decision_manager: auto_execute 失敗路徑補 KM fire-and-forget 寫入
- ai_router: _build_fallback_chain 標記 DEPRECATED 2026-04-28
- tests: test_golden_regression.py 新增 172 行 golden 回歸測試
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-28 15:27:33 +08:00
AWOOOI CD
2e6ae7fe84
chore(cd): deploy 7f200af [skip ci]
2026-04-28 07:14:34 +00:00
Your Name
7f200aff5f
fix(solver): 注入告警 labels 讓 params 模板填充真實值
...
CD Pipeline / build-and-deploy (push) Successful in 10m45s
根因:Solver LLM 不知道 namespace/pod/deployment/instance 真實值,
recommended_actions.params 模板({labels.namespace} 等)填不出來
→ Telegram 顯示 kubectl scale deployment --replicas=(空白)
修復:
- solver.run() 加 incident_labels 參數
- _build_prompt() 把 labels 顯式列出給 LLM 參考
- orchestrator 從 snapshot.alert_info.labels 取出後傳入
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-28 15:05:06 +08:00
AWOOOI CD
b8a330f9e4
chore(cd): deploy c1a1be6 [skip ci]
2026-04-27 12:21:13 +00:00
Your Name
c1a1be61bd
fix(ssh-auto): 主機告警 SSH 自動診斷授權(HostHighCpuLoad 不再卡人工審核)
...
CD Pipeline / build-and-deploy (push) Successful in 9m7s
根因:SSH_MCP_ALLOWED_HOSTS 未設定 → _ssh_execute() 全部攔截
+ auto_approve 只認 kubectl 不認 ssh → 主機告警永遠降級人工
修復:
- ConfigMap: 補 SSH_MCP_ALLOWED_HOSTS 四主機白名單
- alert_rules: HostHighCpuLoad 等從 NO_ACTION 改為 ssh_diagnose 指令
- auto_approve: _has_executable 加入 ssh 開頭識別
- decision_manager: _ssh_execute() 加入 ssh_diagnose 路由
- ssh_provider: 新增 ssh_diagnose tool(ps aux + free -h + df -h,只讀)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 20:13:07 +08:00
Your Name
277808758d
fix(failover): 補 OllamaRoutingResult.health_188 optional 欄位(merge conflict 遺漏)
...
CD Pipeline / build-and-deploy (push) Has been cancelled
stash pop 時 --theirs 覆蓋掉了 health_188 dataclass 欄位定義,
導致 to_dict() 拋出 AttributeError(health_188 只在方法內引用)。
補上 health_188: HealthReport | None = None,37 failover tests ✅
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 20:04:49 +08:00
Your Name
877c2651bf
feat(p3.2.3): provider版本變更Telegram告警 + Gemini quota訊息更新
...
CD Pipeline / build-and-deploy (push) Failing after 1m40s
- FailoverAlerter.alert_provider_version_changed():
- 每個 provider 獨立 dedup key(TTL 3600s),避免頻繁重複告警
- 批次合併通知:同一輪變更一則訊息,標出哪些 provider 版本異動
- 例外由 tracker 層 try/except 攔截,不中斷探測排程
- ModelVersionTracker.run_probe_cycle():
- changed_providers 非空時呼叫 alert_provider_version_changed()
- P3.2.3 整合完成,告警鏈路 probe → 比對 → DB → Telegram 全通
- Gemini quota 告警訊息更新:移除舊的 188 CPU 備援字眼,改為 Nemotron → Claude
- 6 new tests, 1501 passed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 20:00:03 +08:00
Your Name
b6e4e87e57
test(p3.2): provider_version_alerter 單元測試(6 passed)
...
CD Pipeline / build-and-deploy (push) Has been cancelled
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 19:56:51 +08:00
Your Name
ae5e33d254
feat(failover+dispatcher): 補齊 unstaged 服務變更
...
- callback_dispatcher: params 型別放寬支援 numeric
- failover_alerter: alert TTL 修正
- model_version_tracker: 小調整
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 19:56:51 +08:00
Your Name
3e382a4225
fix(telegram): P0 async race + P1 short_id 碰撞 + P0 incident_id 修復
...
- _build_llm_action_buttons 改 async,await setex 在 return 前完成
(消除「按鈕發出→點擊→Redis 未寫完」的 race)
- short_id 從 4 bytes → 8 bytes(16-hex),64-bit 碰撞空間
- payload 加入 incident_id,callback handler 從 payload 還原真實 ID
(修 P0-2:避免 short_id 進 context 造成 KM 學習鏈錯亂)
- Redis 故障與按鈕過期分流回應(P1)
- HTML escape 防 XSS(P2)
- _build_inline_keyboard 改 async,兩個呼叫端加 await
- tests 全部改 @pytest.mark.asyncio + AsyncMock redis
(1495 passed in unit suite)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 19:56:51 +08:00
AWOOOI CD
ded17caca0
chore(cd): deploy a0502b7 [skip ci]
2026-04-27 11:55:33 +00:00
Your Name
a0502b778e
feat(auto-execute): CS3 alertmanager AI path 高信心自動執行(修法3擴展)
...
CD Pipeline / build-and-deploy (push) Successful in 9m41s
- CS3(alertmanager AI path)補入與 CS1 相同的 5 safety gate 自動執行邏輯
- confidence >= 0.85 + !CRITICAL + kubectl非空 + !NO_ACTION + !DESTRUCTIVE
- 使用 _cs3_destr_patterns(from auto_approve)做破壞性指令攔截
- 例外包覆 try/except,不影響主流程
- 新增 test_cs3_auto_execute.py,9 tests 全通過
- CS4(LLM fallback)action=OBSERVE/confidence=0.0 → 不需要 auto-execute,維持現狀
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 19:46:56 +08:00
Your Name
d0c24275d6
fix(incident): Alertmanager 告警補寫 frequency_stats → 歷史統計不再空白
...
CD Pipeline / build-and-deploy (push) Has been cancelled
根因:create_incident_for_approval 建立 Incident 時從未查詢 AnomalyCounter
→ frequency_snapshot 永遠 null → 歷史按鈕顯示「無建立時快照」
signoz/sentry webhook 有寫,Alertmanager 路徑漏掉
修復:建立前 record_anomaly → 頻率快照存入 frequency_stats → PG 持久化
失敗無害(try/except,不阻斷主流程)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 19:41:10 +08:00
AWOOOI CD
0a22f49932
chore(cd): deploy e3bad58 [skip ci]
2026-04-27 08:21:06 +00:00
Your Name
e3bad58842
feat(auto-rate): CS1 LLM 高信心度路徑自動執行(confidence ≥ 0.85)
...
CD Pipeline / build-and-deploy (push) Successful in 9m53s
繼 CS2 rule_engine 後,CS1 LLM 路徑也開啟自動執行:
- confidence >= 0.85 + low/medium risk + kubectl 有值 → auto-execute
- CRITICAL / DESTRUCTIVE_PATTERNS / NO_ACTION → 絕對不執行
- 例外降級到 PENDING,不 crash
- 9 tests 驗收(1469 passed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 16:12:30 +08:00
AWOOOI CD
dfbf3f8f20
chore(cd): deploy a184b82 [skip ci]
2026-04-27 08:08:52 +00:00
Your Name
e5f8d90451
feat(auto-rate): rule_engine 路徑開啟自動執行,預計 42% → 70%+
...
CD Pipeline / build-and-deploy (push) Has been cancelled
修法 3(debugger 建議):CS2 is_rule_based=True + kubectl 有值 + 非 CRITICAL/DESTRUCTIVE → 直接 auto-execute,不建 PENDING record
安全防線(5 層):
- CRITICAL risk → 絕對不自動執行
- _DESTRUCTIVE_PATTERNS 命中 → 絕對不自動執行
- NO_ACTION → 不執行
- kubectl 空字串 → 不執行
- 任何例外 → catch + 降級到 PENDING,不 crash
15 tests 驗收(1487 passed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 16:08:50 +08:00
Your Name
a184b82ed1
feat(webhook): shadow-run auto_approve.evaluate + 補 metadata kwarg
...
CD Pipeline / build-and-deploy (push) Has been cancelled
4 個 webhook call site 問題修復(debugger 根因分析 2026-04-27):
- 補 metadata kwarg → extra_metadata 不再為 NULL(source/confidence_score/is_rule_based/playbook_id)
- shadow-run policy.evaluate() → logger.info 觀測 should_auto_approve
- 不改任何執行決策:status 仍 pending,Telegram 推送不變
- 9 tests 驗收 metadata 非 null + shadow log 格式 + 例外不 propagate
下一步:shadow 觀測 1-2 天後開啟修法 3(rule_based 路徑自動執行)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 16:00:00 +08:00
Your Name
0fd71b3e33
fix(mcp/k8s): _kubectl_scale 補 validate_deployment_exists dry-run
...
CD Pipeline / build-and-deploy (push) Has been cancelled
根因:_kubectl_restart 有 dry-run 驗證,_kubectl_scale 完全沒有
→ gitea(docker-compose,不在 K8s)直接被 kubectl scale 執行
→ Deployment 'gitea' not found in namespace 'awoooi-prod'(INC-20260425-3B6C39)
修復:_kubectl_scale 在執行前加 validate_deployment_exists,
K8s 找不到 deployment 時返回 error 而非繼續執行
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 15:59:37 +08:00
Your Name
c3fa03fc19
fix(solver): 補 AGENT_SOLVER_TIMEOUT_SEC=80 + prompt 禁無腦重啟
...
CD Pipeline / build-and-deploy (push) Has been cancelled
問題1:AGENT_SOLVER_TIMEOUT_SEC 預設 20s K8s 未設 → deepseek-r1:14b 必然
timeout → candidates=[] → action="" → Telegram 顯示「待分析」+「規則分析」
問題2:Solver prompt JSON 範例只有 restart + kubectl top,LLM 模仿範例
→ 所有告警都推重啟,HostDisk/CPU 類應優先診斷+清理
修復:
- K8s 加 AGENT_SOLVER_TIMEOUT_SEC=80(< OPENCLAW_TIMEOUT=120,留 buffer)
- Solver prompt 加根因對應修復規則:HostDisk→df/du/journalctl,CPU→top/ps,
OOM→kubectl logs,禁止「先重啟」
- JSON 範例改為 HostDisk SSH 診斷場景,不再只有 K8s 命令
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 15:51:42 +08:00
Your Name
b432becd4e
fix(failover): 188 完全移出 routing chain,備援只用 Gemini
...
CD Pipeline / build-and-deploy (push) Has been cancelled
統帥鐵律 2026-04-26:
- 唯一 Ollama = 111(M1 Pro Metal 加速)
- 188 CPU-only (0.45 tok/s) 禁止即時回應,移出所有 fallback chain
- 111 HEALTHY → fallback=[Gemini]
- 111 非HEALTHY → primary=Gemini, fallback=[Nemotron, Claude]
- Gemini quota exceeded → Nemotron → Claude(不落 188)
- OllamaRoutingResult 移除 health_188 欄位
- select_provider 只 check 111(不再 asyncio.gather 兩節點)
- 測試全部對齊新規則(1451 passed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 15:47:41 +08:00
Your Name
1b6a4dc14c
fix(k8s): 補 AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=100 救急 step_timeout
...
CD Pipeline / build-and-deploy (push) Has been cancelled
根因:deepseek-r1:14b 推理單題實測 28s,SRE prompt 更長必然 >30s
AGENT_DIAGNOSTICIAN_TIMEOUT_SEC 預設 30s,K8s 沒有覆寫
導致 diagnostician 必然 step_timeout → 信心 20% 降級
修復:K8s 加 AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=100(低於 OPENCLAW_TIMEOUT=120,留 20s buffer)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 15:40:46 +08:00
AWOOOI CD
e0ca1c1f78
chore(cd): deploy ea23972 [skip ci]
2026-04-27 07:30:40 +00:00
Your Name
ea23972f7a
feat(dispatch): B2 LLM 動態 MCP 派發安全閘 + telegram_gateway LLM 按鈕流程
...
CD Pipeline / build-and-deploy (push) Successful in 9m10s
ADR-082 §B2:dispatch_llm_action() 風險閘控 + allowlist + 模板渲染
23 tests pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 15:22:31 +08:00
AWOOOI CD
92a5d94382
chore(cd): deploy f4998b3 [skip ci]
2026-04-27 07:15:37 +00:00
Your Name
f4998b3eee
fix(test): 修 P3.4 governance_agent 加第 5 項 slo_compliance 後既有測試對齊
...
CD Pipeline / build-and-deploy (push) Successful in 10m35s
P3.4 加入 check_slo_compliance 後:
- test_governance_agent::test_all_checks_fail_returns_all_errors: 4→5
- test_wave8_remaining_blockers::TestB8GovernanceFailureAlert: 三測試補 mock
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 15:06:58 +08:00
Your Name
8d6e086254
fix(p3.2): model_version_tracker 改 pure unit test + probe 改善
...
CD Pipeline / build-and-deploy (push) Failing after 2m7s
Engineer 重寫 test_model_version_tracker:
- 用 _make_fake_ctx (asynccontextmanager) 完整 mock get_db_context
- 移除 @pytest.mark.integration(整 class)
- patch probe_all_providers + get_db_context 雙路徑
- 4 testcases 全綠,無真實 PG 依賴
model_version_probe.py 配套改善(match 新 test mock 預期)
Tests: 19 passed (probe 15 + tracker 4)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 14:58:46 +08:00
Your Name
ed205489c1
feat(p3.2-tests+ci-schema): model_version 測試 + CI test_schema 對齊 + Grafana SLO Dashboard
...
CD Pipeline / build-and-deploy (push) Failing after 1m20s
P3.2 配套測試 + CI 環境同步 + ADR-100 Grafana 視覺化:
CI test_schema 補齊(解 1162-1172 阻塞之延伸):
- setup_test_schema.sql 加 ai_provider_version_history 表
- 對齊 production p3_2_provider_version_history.sql(已 K8s exec 上線)
新增測試 (636 行):
- test_model_version_probe.py (387) — Provider 探測單元測試
- test_model_version_tracker.py (249) — Tracker 整合測試
· 4 個 DB-dependent tests 標 @pytest.mark.integration
· 15 unit + 4 integration(unit step 跳過 integration class)
新增配套:
- ai-slo-dashboard.json (496 行) — Grafana 儀表板
· 對應 ADR-100 SLO 規則的 4 大面板:
自主修復成功率 / 飛輪閉環延遲 / 治理事件 / Provider 健康度
修改:
- governance_agent.py +122 行 — SLO 指標暴露 + retrieve metric 整合
Tests: 15 passed (probe + tracker unit), 4 deselected (integration class)
Production 部署狀態:
- p2_decision_fusion_columns.sql ✅ K8s exec 完成(commit c58bdd0c)
- p3_2_provider_version_history.sql ✅ K8s exec 完成(this commit)
- 兩個 production migration 都已上線,CI test_schema 同步補齊
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 14:57:16 +08:00
Your Name
025a493f06
feat(p3.2+adr-100): Model Version Tracker + SLO 自治 + KB rot cleaner
...
run-migration / migrate (push) Failing after 12s
CD Pipeline / build-and-deploy (push) Has been cancelled
Wave 8 P3.2 模型版本追蹤 + ADR-100 SLO 自我治理 + 配套:
P3.2 — Model Version Tracking:
- model_version_probe.py (268 行) — 探測 Ollama / OpenRouter 等 provider 的 model version
- model_version_tracker.py (101 行) — 對齊 PG provider_version_history 表
- migrations/p3_2_provider_version_history.sql + rollback — 25 行 schema
- db/models.py +32 行 — ProviderVersionHistory ORM
ADR-100 — AI 自主化 SLO:
- docs/adr/ADR-100-ai-autonomous-slo.md (167 行) — 飛輪 SLO 設計與閾值
- ops/monitoring/slo-rules.yml (254 行) — Prometheus SLO recording rules + alerts
- ops/monitoring/tests/test_slo_rules.yaml (242 行) — promtool unit tests
整合修改:
- main.py +72 行 — Lifespan 啟動 model_version_probe + KB rot cleaner schedule
- gitea_webhook.py +45 行 — webhook 接收 model 版本變化通知
- ci_auto_repair.py / evidence_snapshot.py / pre_decision_investigator.py — 配合接線
新測試:
- test_kb_rot_cleaner_schedule.py (120 行) — 9 tests pass
- test_slo_rules.yaml — promtool 驗收
Tests: 9 passed (test_kb_rot_cleaner_schedule)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Multiple Engineers (P3.2 + ADR-100) <noreply@anthropic.com >
2026-04-27 14:54:19 +08:00
Your Name
9908fdf50d
feat(p3.1-t2-patha): DiagnosisAggregator 路徑 A + Solver F4 critical reject + 對齊測試
...
CD Pipeline / build-and-deploy (push) Failing after 1m59s
Wave 8 P3.1-T2 PathA 啟用 + Solver F4 安全強化 + test 對齊:
PathA — DiagnosisAggregator 信號分類層補 PDI:
- ENABLE_DIAGNOSIS_AGGREGATOR default=False → True
· PathA 純信號分類層(OOMKilled/CrashLoop 等業務邏輯)
· 不重複呼叫 K8s/SignOz API(只取 PDI 已收集的 raw 資料)
· 安全 default on — 純邏輯處理,無外部依賴重疊
- diagnosis_aggregator.py +155 行(PathA 實作)
- pre_decision_investigator.py 已接 (commit 3a2cd151 )
F4 — Solver critical risk reject:
- solver_agent.py: _validate_recommended_action 拒絕 risk=critical
· 鐵律:critical 動作必須走人工審批,不可變 Telegram 按鈕
· log warning + return None(被 _extract 過濾掉)
- _extract_recommended_actions 改返回 (list, status_str) tuple
· status="ok"/"empty"/"all_invalid" 供呼叫端決策
- protocol.py +16 / metrics.py +9 / ai_router.py +18 — 配套 metric + protocol field
測試對齊:
- test_solver_recommended_actions.py 拆 test_all_valid → low/medium/high accepted +
test_critical_rejected
- result tuple unpack: result, _ = _extract_recommended_actions(...)
- test_diagnosis_aggregator_stub.py: feature flag default 改 True 對齊 PathA
Tests: 51 passed (solver 28 + aggregator 16 + router fallback 8)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Multiple Engineers (Wave 8 P3.1-T2 PathA + F4) <noreply@anthropic.com >
2026-04-27 14:42:29 +08:00
Your Name
f09a8f56a9
fix(ci): test_schema 加 P2.1 fusion 欄位 — 解 CI 1162-1172 阻塞
...
CD Pipeline / build-and-deploy (push) Has been cancelled
Production PG migration 已上線(commit c58bdd0c),但 CI 用獨立 docker pgvector
test container(pg-test-b5),由 setup_test_schema.sql 初始化 → 無 fusion 欄位
→ test_b5_core_flows.py 整合測試失敗於 composite_score column does not exist。
修法:把 P2.1 ALTER TABLE 加入 setup_test_schema.sql(idempotent IF NOT EXISTS)
新增(對齊 production p2_decision_fusion_columns.sql):
- composite_score REAL
- complexity_tier VARCHAR(16) + CHECK ('low','medium','high','critical')
- decision_fusion_details JSONB
partial index 不需要在 test schema(B5 整合測試不依賴 index)。
DO $$ block 處理 CHECK constraint 因 PG 不支援 ADD CONSTRAINT IF NOT EXISTS。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 14:39:06 +08:00
Your Name
fb130c9a28
feat(p3.1-t2): DiagnosisAggregator stub tests + sanitization 補強 + metrics 補欄
...
CD Pipeline / build-and-deploy (push) Failing after 2m16s
Wave 8 P3.1-T2 後續補測 + 配套:
新增測試:
- test_diagnosis_aggregator_stub.py (238 行) — 15 tests
· stub fixture 驗證 _collect_diagnosis_aggregator 接線
· feature flag default off 不呼叫
· timeout 邊界 / exception fail-soft
修改:
- core/metrics.py +23 — 新增 DiagnosisAggregator 相關 Prometheus 指標
- sanitization_service.py +24 — 補強 prompt sanitize 邊界(vuln #4 配套)
- RUNBOOK-AGENT-STEP-LATENCY.md / agent_step_latency_rules.yaml — 微調
Tests: 15 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:30:26 +08:00
Your Name
c58bdd0c38
chore(cd-trigger): production PG migration p2_decision_fusion_columns 已執行
...
統帥授權執行於 192.168.0.188:5432/awoooi_prod via K8s pod exec:
- composite_score REAL
- complexity_tier VARCHAR(16) + CHECK ('low','medium','high','critical')
- decision_fusion_details JSONB
- ix_approval_composite_score (partial, WHERE composite_score IS NOT NULL)
- ix_approval_complexity_tier (partial, WHERE complexity_tier IS NOT NULL)
Pre-existing CI integration test 阻塞解,全部 25+ commits 應一次部署。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:29:57 +08:00
Your Name
9a711278f7
test(p3.1-t2): Sentry Webhook 簽章驗證 dedicated tests
...
CD Pipeline / build-and-deploy (push) Failing after 1m23s
對應 commit 3a2cd151 的 SentryWebhookService.verify_sentry_signature 整合驗證。
Tests: 18 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:24:59 +08:00
Your Name
2b39558492
test(governance): trust_drift_watchdog dedicated tests
...
CD Pipeline / build-and-deploy (push) Has been cancelled
P2.2 governance 補測:trust_drift watchdog 9 個整合測試。
Tests: 9 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:24:37 +08:00
Your Name
3a2cd15144
feat(p3.1-t2): Tier-2 三服務感知強化 — Sentry 簽章 + DiagnosisAggregator + Solver actions test
...
CD Pipeline / build-and-deploy (push) Has been cancelled
Wave 8 P3.1-T2 三項感知強化(多 engineer 補完):
Sentry Webhook 簽章驗證:
- sentry_webhook.py: 接入 SentryWebhookService.verify_sentry_signature()
- 拒絕無效 sentry-hook-signature → 401 → 防偽造攻擊
DiagnosisAggregator Pod 深診斷整合:
- pre_decision_investigator.py: 新增 _collect_diagnosis_aggregator()
- ENABLE_DIAGNOSIS_AGGREGATOR feature flag 守衛(default=False)
- evidence_snapshot.py: extra_diagnosis 欄位 + build_summary 顯示
- timeout=3.0s + try/except 隔離(fail-soft)
- Conservative 策略:待重疊分析確認 vs PreDecisionInvestigator 不重複
config.py:
- 新增 ENABLE_DIAGNOSIS_AGGREGATOR Field(default=False,K8s ConfigMap 動態啟用)
Solver B1 補測(commit 7c726ebc 對應):
- test_solver_recommended_actions.py — 20 tests + 3 skipped
- 驗證結構化 recommended_actions(北極星 §1.1 修復多樣性 ≥ 40%)
- LLM 失敗 graceful degraded(candidates=[], degraded=True)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Multiple Engineers (Wave 8 P3.1-T2) <noreply@anthropic.com >
2026-04-27 08:24:15 +08:00
Your Name
6de10cb073
test(wave8-blockers): 4 餘項 BLOCKER 修復驗收(vuln #4 + B14 + B25/B26 + B8)
...
CD Pipeline / build-and-deploy (push) Has been cancelled
確認 critic + debugger + vuln-verifier 報告中尚未驗收的 4 修復都已實裝在 production,
並補對應 dedicated tests:
vuln #4 — fusion prompt injection 防禦:
- score_with_elephant 內 _sanitize 剔除控制字元 + 截長至 max_len
- alert_name(100) / evidence(...) / proposal(300) 三層 sanitize
- 驗證:1000 個 'A' 攻擊 payload → prompt 內 'A' < 200,控制字元 \\x00\\x1b\\x02 全剔除
debugger B14 — Gemini quota fail-closed:
- ollama_failover_manager._check_gemini_quota except branch
- Redis 異常時 return False(非 fail-open),費用安全 > 服務可用性
- best-effort 呼叫 alert_gemini_quota_exceeded 通知運維
debugger B25/B26 — auto_repair drain_pending_tasks:
- AutoRepairService._pending_tasks (set) + drain_pending_tasks(timeout=60.0)
- main.py shutdown 已接 _repair_svc.drain_pending_tasks() 呼叫
- K8s rolling restart 時 fire-and-forget tasks 不丟失
debugger B8 — governance ≥3 failures alert:
- run_self_check 後聚合 failed_checks
- ≥3 項失敗 → self._alert("governance_self_failure", ...) 觸發
- payload 含 failed_checks list + total_checks=4 + errors dict
Tests: 10/10 PASSED (vuln 3 + B14 2 + drain 2 + governance 3)
Note: 此 commit 純補測,所有 4 修復代碼上 commit 已 in production
仍待: 1167+ CD runs 確認 deploy 成功
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:22:47 +08:00
Your Name
7c726ebc1c
fix(b1): Solver Agent 結構化動作 — 北極星 §1.1 修復多樣性 ≥ 40%
...
CD Pipeline / build-and-deploy (push) Failing after 2m22s
INC-20260425 衍生修復 — Solver 拒絕 rule-based mock 兜底:
原設計缺陷:
- LLM 失敗時 → rule-based mock 推 RESTART 兜底
- 違反北極星 §1.1:修復多樣性 ≥ 40%(不能寫死同一指令)
新設計:
- LLM 失敗 → graceful degraded(candidates=[], recommended_actions=[], degraded=True)
- 禁止 rule-based mock / hardcode RESTART
- 新增 recommended_actions 結構化 MCP 動作清單
· 供 B3 Telegram 按鈕動態生成
· YAML 規則庫驅動,非寫死
- 新增 yaml + Path import 載入動作模板庫
向下相容:
- 既有 candidates / blast_radius 邏輯不變
- 新增欄位 recommended_actions 為 optional list
Tests: 8 passed (solver 相關全綠)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Claude Sonnet 4.6 (B1 北極星 §1.1) <noreply@anthropic.com >
2026-04-27 08:18:38 +08:00
Your Name
21977004e7
test(p3.1-t1): test_p3_tier1_integrations 對應 model_rollback + resource_resolver 整合
...
CD Pipeline / build-and-deploy (push) Has been cancelled
P3.1-T1 接線測試(補 commit 123d9c8a 的 dedicated tests):
- model_rollback_service.check() 在 offline_replay 後被呼叫
- resource_resolver.resolve() 在 approval_execution 解析 kubectl 後被呼叫
- exception fail-soft 路徑驗證
- RESOURCE_RESOLVE_TOTAL counter 各 label
Tests: 12 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:17:59 +08:00
Your Name
123d9c8a2e
fix(p3.1-t1): 三 Tier-1 服務整合 — model_rollback_service + resource_resolver
...
CD Pipeline / build-and-deploy (push) Has been cancelled
P3.1-T1 接線兩個既有服務到主流程:
offline_replay_service.py — model_rollback_service 整合:
- 回放事件寫入治理 DB 後,觸發 ModelRollbackService.check() 衰退偵測
- feature flag 由 model_rollback_service 自行判斷(AIOPS_P6_GOVERNANCE_ENABLED)
- retrain_recommended → log warning 含 streak / absolute_floor / conservative_mode
- exception fail-soft(不阻斷 replay 主流程)
approval_execution.py — resource_resolver 整合:
- kubectl 指令解析後,動態驗證資源是否存在於 K8s
- 若 resolved_name != raw_name → log + apply normalized name
- 若不存在但有 candidates → log warning + suggestions(不攔截執行,只記錄)
- exception fail-soft(不阻斷主流程)
- RESOURCE_RESOLVE_TOTAL Prometheus counter labels: hit/suggestion/miss/error
Tests: 後端 1303 collected(無回歸),對應 dedicated 測試在前次 commit 已寫
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Claude Sonnet 4.6 (P3.1-T1) <noreply@anthropic.com >
2026-04-27 08:17:04 +08:00
Your Name
fefe4c21cd
fix(inc-20260425): A1+A2 後續 — Solver/Critic timeout + auto_repair 接線 + Runbook + Grafana
...
CD Pipeline / build-and-deploy (push) Has been cancelled
延續 595629c0 INC-20260425 修復,補三段 Agent + 全鏈路觀測:
A1 後續 — Solver/Critic 三段 timeout 接線:
- solver_agent.py: AGENT_SOLVER_TIMEOUT_SEC=20.0(env override)
- critic_agent.py: AGENT_CRITIC_TIMEOUT_SEC=15.0(env override)
- protocol.py: 三 Agent 共用 observe_agent_step() 包裹呼叫
· success/timeout/error outcome label
· histogram 寫入 aiops_agent_step_duration_seconds
A2 後續 — auto_repair_service 改用 _diagnose_fallback_chain:
- auto_repair_service.py +46 行 — 切換 DIAGNOSE 路由到新 chain(NEMO→GEMINI→CLAUDE)
- 完全避開 Ollama CPU 238s 二次 timeout
新增 metrics:
- core/metrics.py +59 行 — 配合 observe_agent_step 的 histogram bucket + label cardinality
新增測試 (862 行):
- test_agent_step_timeouts.py (475) — 三 Agent 各 timeout 邊界 + outcome label
- test_ai_router_diagnose_fallback.py (387) — _diagnose_fallback_chain 正確序
新增配套:
- docs/runbooks/RUNBOOK-AGENT-STEP-LATENCY.md (350) — INC 故障排查 + 觀測指引
- ops/monitoring/grafana/agent_step_latency_rules.yaml (160)
· 三 Agent histogram alert rules(p99 > timeout 80% → warning)
驗收: 33 tests pass (test_agent_step_timeouts 22 + test_ai_router_diagnose_fallback 11)
INC-20260425 雙修總工作量(595629c0 + 此 commit):
· 5 個 service/agent 檔修改
· 1 個新 observability 模組
· 4 個新測試/配套檔
· 1372+187 = 1559 行新增
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Claude Sonnet 4.6 (INC-20260425 後續) <noreply@anthropic.com >
2026-04-27 08:15:53 +08:00
Your Name
595629c013
fix(inc-20260425): A1 三段 Agent timeout 拆分 + A2 DIAGNOSE 移除 Ollama
...
CD Pipeline / build-and-deploy (push) Has been cancelled
INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20% 根因雙修(統帥批准 A+B):
A1 — 三段 Agent step timeout 拆分(北極星 §1.2 Observable by Default):
- diagnostician_agent.py: PHASE2_STEP_TIMEOUT_SEC=20.0 共用值 → 拆三段
· AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=30.0(NIM 主吃口,最大 prompt + 多假設)
· AGENT_SOLVER_TIMEOUT_SEC=20.0(後續 commit 接線)
· AGENT_CRITIC_TIMEOUT_SEC=15.0(後續 commit 接線)
· env override 支援,K8s ConfigMap 動態調整不需 rebuild
· 保留 PHASE2_STEP_TIMEOUT_SEC alias(DEPRECATED,下 sprint 移除)
- observability/agent_step_metrics.py (58 行) — 新模組:
· aiops_agent_step_duration_seconds Histogram
· observe_agent_step() helper 統一三 Agent 呼叫點
· outcome label ∈ {success, timeout, error}
· agent label ∈ {diagnostician, solver, critic}
A2 — ai_router DIAGNOSE chain 移除 Ollama:
- ai_router.py v4.4 by Claude Sonnet 4.6
· 新增 _diagnose_fallback_chain: NEMO → GEMINI → CLAUDE
· Ollama 永久排除於此 chain(CPU-only 實測 238s,二次 timeout 必爆)
· 新增 aiops_diagnose_fallback_total Prometheus metric
- 根因: NIM timeout 後 fallback 到 Ollama deepseek-r1:14b CPU 238s
→ 二次 timeout → degraded confidence=0.2
Wave8-X2 整合測試補正:
- test_ollama_failover_manager.py: TestSelectProvider 補 mock _check_gemini_quota
原 test 期望 OFFLINE→Gemini,但 quota fail-closed 後沒 mock 會被切到 188
繞過 quota check 後驗純路由邏輯 → 37/37 PASS
Tests: 37 passed (test_ollama_failover_manager 全部)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Claude Sonnet 4.6 (Wave 8 INC-20260425) <noreply@anthropic.com >
2026-04-27 08:15:10 +08:00
Your Name
1ab6786ce3
feat(ops): Ollama 容災 Runbook + Grafana 儀表板 + Consensus K8s ConfigMap patch
...
run-migration / migrate (push) Failing after 13s
CD Pipeline / build-and-deploy (push) Failing after 2m1s
Wave 6 P2.3 ops 配套 + tool-expert 部署文件:
新增:
- docs/runbooks/RUNBOOK-OLLAMA-FAILOVER.md (240 行)
· 三大鐵律驗證步驟(自動切 Gemini / 自動切回 / quota 熔斷)
· failover/recovery 完整 SOP
· 故障排查清單(Ollama 111/188 不通、Gemini quota 超發等)
- ops/monitoring/grafana/dashboards/ollama_failover.json (295 行)
· 4 panel:current primary / failover events / quota usage / health status
· 對應 P2.3 metrics: OLLAMA_FAILOVER_TRIGGERED_TOTAL / GEMINI_DAILY_CALL_COUNT
- k8s/awoooi-prod/04-configmap.yaml.patch-consensus
· ENABLE_12AGENT_CONSENSUS / ENABLE_AIOPS_P2_FUSION feature flag patch
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: tool-expert agent (Wave 6) <noreply@anthropic.com >
2026-04-27 08:11:40 +08:00
Your Name
1096da12ae
feat(p2.5): aiops 時序前端面板 — Incident 6 階段視覺化
...
Wave 6 P2.5 frontend-designer 工業級視覺化(拒絕 AI slop):
新增(1824 行):
- apps/web/src/app/[locale]/aiops/timeline/page.tsx
- apps/web/src/components/aiops/timeline/
· AiopsTimelinePanel.tsx (413) — 主面板組件
· TimelineStage.tsx (279) — 6 階段時序卡片
· TimelineStageDetails.tsx (359) — 階段細節展開
· EvidenceViewer.tsx (144) — Evidence Snapshot 檢視
· TimelineFilter.tsx (109) — incident_id / severity / 時段 過濾器
· types.ts (118) — TS 型別定義
· mock-data.ts (357) — 開發 mock fallback
· index.ts (7) — barrel export
- i18n: messages/en.json + messages/zh-TW.json — Timeline 翻譯
設計原則:
- 拒絕 AI slop(無泛用 emoji/漸層,採工業 dashboard 風格)
- 後端 endpoint 接通 /api/v1/aiops/timeline(critic B4 修復)
- mock 模式 fallback 防 endpoint 暫時不可達
對應後端: a3b4595e(aiops_timeline.py + aiops_timeline_service.py)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: frontend-designer agent (Wave 6) <noreply@anthropic.com >
2026-04-27 08:11:40 +08:00