Your Name
6eb33594c2
docs(logbook): T0 12-Agent 全景驗證紀錄
...
承接前段 session wave2 (commit 143c15f0noreply@anthropic.com >
2026-04-29 10:44:39 +08:00
Your Name
c22e5f334e
feat(km): P1-1 KMWriter 統一契約 + 5 caller 切換 + M4 反查鏈補齊
...
12-Agent 全景診斷揪出 KM 寫入鏈路 5 條入口無統一契約，fire-and-forget
在 Pod recycle 時會丟失條目。本次抽 KMWriter 強制 7 條契約。
## 7 條契約強制
1. 同步底線：強制 await asyncio.wait_for(timeout)
2. 重試：3 次指數退避 1s/2s/4s（OperationalError / 網路類例外）
3. 失敗回收：3 次後寫 Redis DLQ km:dlq + log
4. 觀測：structlog event + 預留 metric hook（P1-3 補 emitter）
5. 冪等：incident_id + path_type 為 unique key
6. 禁止吞例外：except 必須 log + raise/DLQ
7. M4 反查鏈：payload 含 approval_id 時自動填 related_approval_id 並回填 Path A
## Caller 切換（5 條入口統一介面）
- incident_service.py:1086 Path A（KB extractor + km_conversion）
- approval_execution.py:771 Path B-人工
- decision_manager.py:2178 Path B-自動成功（消除跨類私有方法調用 M1）
- decision_manager.py:2200 Path B-自動失敗（修 B2 早期吞例外）
- playbook_service.py:210 PlaybookKM（兩份 T0 報告都漏的第三條）
## M4 反查鏈補齊
- knowledge.py + models.py: 補 related_approval_id ORM 欄位
- 對齊 phase26_incident_km_integration.sql:20 schema（partial index 已存在）
- approval↔KM 雙向反查鏈完整（dual-path 縫合線）
## Feature Flag (rollback 保險)
- KM_WRITE_AWAIT=true (default): await + timeout + DLQ 強制
- KM_WRITE_AWAIT=false: fire-and-forget（舊行為）
## 測試
- apps/api/tests/test_km_writer.py: 18 測試全綠
覆蓋 success / timeout / retry / DLQ / 冪等 / KMWriteError /
on_failure=raise / 反查鏈回填
- 1552 unit tests 全綠（無回歸）
## 驗收
飛輪閉環核心 — KM 寫入不再靜默丟失，AI 學習鏈不斷裂。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-29 10:44:39 +08:00
Your Name
715dc3cb91
fix(observability): P0 假警報止血 + ConfigMap drift 對齊 + 治理工具
...
12-Agent 全景診斷觸發的 P0/P1 觀測層修復。
## P0 假警報止血（4 SLO 雪崩根因）
- governance_agent.py:306 — 空 result 不再 fallback 0.0，改 continue + log warning
根因：Prometheus 查無資料（emitter 未實作 / rule 未部署）被誤判為 SLO=0
必觸發 violated=True 噴 4 條假告警
## P0 鬼魂按鈕守門
- telegram_gateway.py:1654 — LLM 動態按鈕 Redis 失敗時 btn_list.clear()
first_row（批准/拒絕，HMAC nonce 無狀態）由 caller 1488 永遠保留
feedback_no_ghost_buttons.md 三缺一鐵律對齊
## ConfigMap drift 修復（3 處）
- config.py:683 PROMETHEUS_URL: 188→110（drift checker 揪出 = SPF-4 部分根因）
- config.py:705 ARGOCD_URL: 125→121（T0 G3 已知）
- config.py:375 AI_FALLBACK_ORDER: 補 nvidia 對齊 ConfigMap
## P1 Alertmanager 升級（amtool SUCCESS）
- ops/alertmanager/alertmanager.yml: deprecated → v0.27+ 新語法
- match/match_re → matchers
- source_match/target_match → source_matchers/target_matchers
- group_by 加 team label（防 SLO 雪崩 4 條同秒推）
- PostgreSQL/Redis inhibit 補 equal: ['instance']（防爆炸抑制）
- 新增 3 組因果抑制：
- OllamaInstanceDown → SLO_*/AI_*（30 分鐘）
- KMConverterDown → SLO_KMGrowthRate*
- SLO_*_FastBurn → SLO_*_(Medium|Slow)Burn
## 治理工具落地
- scripts/check_config_drift.py: ConfigMap vs code default drift 檢測
揪出 PROMETHEUS_URL drift 是 SPF-4 根因（governance_agent 連 188 而非 110）
- scripts/health_check_session.sh: 11 服務 + 4 SSH + drift + git 全景驗證
## 驗證
- 1552 unit tests 全綠
- amtool check-config SUCCESS（8 inhibit_rules / 2 receivers）
- drift checker 4 欄位全對齊
- health check 11 服務全可達
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-29 10:44:39 +08:00
AWOOOI CD
20009cddcf
chore(cd): deploy 143c15f [skip ci]
2026-04-28 07:36:19 +00:00
Your Name
143c15f052
feat(wave2+km): LLM 動態按鈕啟用 + KM 自動寫入 + AI Router dead code 標記
...
CD Pipeline / build-and-deploy (push) Successful in 9m52s
- ConfigMap: USE_LLM_DYNAMIC_BUTTONS=true（B2/B3/B4 handler 全就緒）
- decision_manager: auto_execute 失敗路徑補 KM fire-and-forget 寫入
- ai_router: _build_fallback_chain 標記 DEPRECATED 2026-04-28
- tests: test_golden_regression.py 新增 172 行 golden 回歸測試
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-28 15:27:33 +08:00
AWOOOI CD
2e6ae7fe84
chore(cd): deploy 7f200af [skip ci]
2026-04-28 07:14:34 +00:00
Your Name
7f200aff5f
fix(solver): 注入告警 labels 讓 params 模板填充真實值
...
CD Pipeline / build-and-deploy (push) Successful in 10m45s
根因：Solver LLM 不知道 namespace/pod/deployment/instance 真實值，
recommended_actions.params 模板（{labels.namespace} 等）填不出來
→ Telegram 顯示 kubectl scale deployment --replicas=（空白）
修復：
- solver.run() 加 incident_labels 參數
- _build_prompt() 把 labels 顯式列出給 LLM 參考
- orchestrator 從 snapshot.alert_info.labels 取出後傳入
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-28 15:05:06 +08:00
AWOOOI CD
b8a330f9e4
chore(cd): deploy c1a1be6 [skip ci]
2026-04-27 12:21:13 +00:00
Your Name
c1a1be61bd
fix(ssh-auto): 主機告警 SSH 自動診斷授權（HostHighCpuLoad 不再卡人工審核）
...
CD Pipeline / build-and-deploy (push) Successful in 9m7s
根因：SSH_MCP_ALLOWED_HOSTS 未設定 → _ssh_execute() 全部攔截
+ auto_approve 只認 kubectl 不認 ssh → 主機告警永遠降級人工
修復：
- ConfigMap: 補 SSH_MCP_ALLOWED_HOSTS 四主機白名單
- alert_rules: HostHighCpuLoad 等從 NO_ACTION 改為 ssh_diagnose 指令
- auto_approve: _has_executable 加入 ssh 開頭識別
- decision_manager: _ssh_execute() 加入 ssh_diagnose 路由
- ssh_provider: 新增 ssh_diagnose tool（ps aux + free -h + df -h，只讀）
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 20:13:07 +08:00
Your Name
277808758d
fix(failover): 補 OllamaRoutingResult.health_188 optional 欄位（merge conflict 遺漏）
...
CD Pipeline / build-and-deploy (push) Has been cancelled
stash pop 時 --theirs 覆蓋掉了 health_188 dataclass 欄位定義，
導致 to_dict() 拋出 AttributeError（health_188 只在方法內引用）。
補上 health_188: HealthReport | None = None，37 failover tests ✅
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 20:04:49 +08:00
Your Name
877c2651bf
feat(p3.2.3): provider版本變更Telegram告警 + Gemini quota訊息更新
...
CD Pipeline / build-and-deploy (push) Failing after 1m40s
- FailoverAlerter.alert_provider_version_changed()：
- 每個 provider 獨立 dedup key（TTL 3600s），避免頻繁重複告警
- 批次合併通知：同一輪變更一則訊息，標出哪些 provider 版本異動
- 例外由 tracker 層 try/except 攔截，不中斷探測排程
- ModelVersionTracker.run_probe_cycle()：
- changed_providers 非空時呼叫 alert_provider_version_changed()
- P3.2.3 整合完成，告警鏈路 probe → 比對 → DB → Telegram 全通
- Gemini quota 告警訊息更新：移除舊的 188 CPU 備援字眼，改為 Nemotron → Claude
- 6 new tests, 1501 passed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 20:00:03 +08:00
Your Name
b6e4e87e57
test(p3.2): provider_version_alerter 單元測試（6 passed）
...
CD Pipeline / build-and-deploy (push) Has been cancelled
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 19:56:51 +08:00
Your Name
ae5e33d254
feat(failover+dispatcher): 補齊 unstaged 服務變更
...
- callback_dispatcher: params 型別放寬支援 numeric
- failover_alerter: alert TTL 修正
- model_version_tracker: 小調整
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 19:56:51 +08:00
Your Name
3e382a4225
fix(telegram): P0 async race + P1 short_id 碰撞 + P0 incident_id 修復
...
- _build_llm_action_buttons 改 async，await setex 在 return 前完成
（消除「按鈕發出→點擊→Redis 未寫完」的 race）
- short_id 從 4 bytes → 8 bytes（16-hex），64-bit 碰撞空間
- payload 加入 incident_id，callback handler 從 payload 還原真實 ID
（修 P0-2：避免 short_id 進 context 造成 KM 學習鏈錯亂）
- Redis 故障與按鈕過期分流回應（P1）
- HTML escape 防 XSS（P2）
- _build_inline_keyboard 改 async，兩個呼叫端加 await
- tests 全部改 @pytest.mark.asyncio + AsyncMock redis
（1495 passed in unit suite）
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 19:56:51 +08:00
AWOOOI CD
ded17caca0
chore(cd): deploy a0502b7 [skip ci]
2026-04-27 11:55:33 +00:00
Your Name
a0502b778e
feat(auto-execute): CS3 alertmanager AI path 高信心自動執行（修法3擴展）
...
CD Pipeline / build-and-deploy (push) Successful in 9m41s
- CS3（alertmanager AI path）補入與 CS1 相同的 5 safety gate 自動執行邏輯
- confidence >= 0.85 + !CRITICAL + kubectl非空 + !NO_ACTION + !DESTRUCTIVE
- 使用 _cs3_destr_patterns（from auto_approve）做破壞性指令攔截
- 例外包覆 try/except，不影響主流程
- 新增 test_cs3_auto_execute.py，9 tests 全通過
- CS4（LLM fallback）action=OBSERVE/confidence=0.0 → 不需要 auto-execute，維持現狀
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 19:46:56 +08:00
Your Name
d0c24275d6
fix(incident): Alertmanager 告警補寫 frequency_stats → 歷史統計不再空白
...
CD Pipeline / build-and-deploy (push) Has been cancelled
根因：create_incident_for_approval 建立 Incident 時從未查詢 AnomalyCounter
→ frequency_snapshot 永遠 null → 歷史按鈕顯示「無建立時快照」
signoz/sentry webhook 有寫，Alertmanager 路徑漏掉
修復：建立前 record_anomaly → 頻率快照存入 frequency_stats → PG 持久化
失敗無害（try/except，不阻斷主流程）
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 19:41:10 +08:00
AWOOOI CD
0a22f49932
chore(cd): deploy e3bad58 [skip ci]
2026-04-27 08:21:06 +00:00
Your Name
e3bad58842
feat(auto-rate): CS1 LLM 高信心度路徑自動執行（confidence ≥ 0.85）
...
CD Pipeline / build-and-deploy (push) Successful in 9m53s
繼 CS2 rule_engine 後，CS1 LLM 路徑也開啟自動執行：
- confidence >= 0.85 + low/medium risk + kubectl 有值 → auto-execute
- CRITICAL / DESTRUCTIVE_PATTERNS / NO_ACTION → 絕對不執行
- 例外降級到 PENDING，不 crash
- 9 tests 驗收（1469 passed）
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 16:12:30 +08:00
AWOOOI CD
dfbf3f8f20
chore(cd): deploy a184b82 [skip ci]
2026-04-27 08:08:52 +00:00
Your Name
e5f8d90451
feat(auto-rate): rule_engine 路徑開啟自動執行，預計 42% → 70%+
...
CD Pipeline / build-and-deploy (push) Has been cancelled
修法 3（debugger 建議）：CS2 is_rule_based=True + kubectl 有值 + 非 CRITICAL/DESTRUCTIVE → 直接 auto-execute，不建 PENDING record
安全防線（5 層）：
- CRITICAL risk → 絕對不自動執行
- _DESTRUCTIVE_PATTERNS 命中 → 絕對不自動執行
- NO_ACTION → 不執行
- kubectl 空字串 → 不執行
- 任何例外 → catch + 降級到 PENDING，不 crash
15 tests 驗收（1487 passed）
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 16:08:50 +08:00
Your Name
a184b82ed1
feat(webhook): shadow-run auto_approve.evaluate + 補 metadata kwarg
...
CD Pipeline / build-and-deploy (push) Has been cancelled
4 個 webhook call site 問題修復（debugger 根因分析 2026-04-27）：
- 補 metadata kwarg → extra_metadata 不再為 NULL（source/confidence_score/is_rule_based/playbook_id）
- shadow-run policy.evaluate() → logger.info 觀測 should_auto_approve
- 不改任何執行決策：status 仍 pending，Telegram 推送不變
- 9 tests 驗收 metadata 非 null + shadow log 格式 + 例外不 propagate
下一步：shadow 觀測 1-2 天後開啟修法 3（rule_based 路徑自動執行）
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 16:00:00 +08:00
Your Name
0fd71b3e33
fix(mcp/k8s): _kubectl_scale 補 validate_deployment_exists dry-run
...
CD Pipeline / build-and-deploy (push) Has been cancelled
根因：_kubectl_restart 有 dry-run 驗證，_kubectl_scale 完全沒有
→ gitea（docker-compose，不在 K8s）直接被 kubectl scale 執行
→ Deployment 'gitea' not found in namespace 'awoooi-prod'（INC-20260425-3B6C39）
修復：_kubectl_scale 在執行前加 validate_deployment_exists，
K8s 找不到 deployment 時返回 error 而非繼續執行
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 15:59:37 +08:00
Your Name
c3fa03fc19
fix(solver): 補 AGENT_SOLVER_TIMEOUT_SEC=80 + prompt 禁無腦重啟
...
CD Pipeline / build-and-deploy (push) Has been cancelled
問題1：AGENT_SOLVER_TIMEOUT_SEC 預設 20s K8s 未設 → deepseek-r1:14b 必然
timeout → candidates=[] → action="" → Telegram 顯示「待分析」+「規則分析」
問題2：Solver prompt JSON 範例只有 restart + kubectl top，LLM 模仿範例
→ 所有告警都推重啟，HostDisk/CPU 類應優先診斷+清理
修復：
- K8s 加 AGENT_SOLVER_TIMEOUT_SEC=80（< OPENCLAW_TIMEOUT=120，留 buffer）
- Solver prompt 加根因對應修復規則：HostDisk→df/du/journalctl，CPU→top/ps，
OOM→kubectl logs，禁止「先重啟」
- JSON 範例改為 HostDisk SSH 診斷場景，不再只有 K8s 命令
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 15:51:42 +08:00
Your Name
b432becd4e
fix(failover): 188 完全移出 routing chain，備援只用 Gemini
...
CD Pipeline / build-and-deploy (push) Has been cancelled
統帥鐵律 2026-04-26：
- 唯一 Ollama = 111（M1 Pro Metal 加速）
- 188 CPU-only (0.45 tok/s) 禁止即時回應，移出所有 fallback chain
- 111 HEALTHY → fallback=[Gemini]
- 111 非HEALTHY → primary=Gemini, fallback=[Nemotron, Claude]
- Gemini quota exceeded → Nemotron → Claude（不落 188）
- OllamaRoutingResult 移除 health_188 欄位
- select_provider 只 check 111（不再 asyncio.gather 兩節點）
- 測試全部對齊新規則（1451 passed）
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 15:47:41 +08:00
Your Name
1b6a4dc14c
fix(k8s): 補 AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=100 救急 step_timeout
...
CD Pipeline / build-and-deploy (push) Has been cancelled
根因：deepseek-r1:14b 推理單題實測 28s，SRE prompt 更長必然 >30s
AGENT_DIAGNOSTICIAN_TIMEOUT_SEC 預設 30s，K8s 沒有覆寫
導致 diagnostician 必然 step_timeout → 信心 20% 降級
修復：K8s 加 AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=100（低於 OPENCLAW_TIMEOUT=120，留 20s buffer）
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com >
2026-04-27 15:40:46 +08:00
AWOOOI CD
e0ca1c1f78
chore(cd): deploy ea23972 [skip ci]
2026-04-27 07:30:40 +00:00
Your Name
ea23972f7a
feat(dispatch): B2 LLM 動態 MCP 派發安全閘 + telegram_gateway LLM 按鈕流程
...
CD Pipeline / build-and-deploy (push) Successful in 9m10s
ADR-082 §B2：dispatch_llm_action() 風險閘控 + allowlist + 模板渲染
23 tests pass
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 15:22:31 +08:00
AWOOOI CD
92a5d94382
chore(cd): deploy f4998b3 [skip ci]
2026-04-27 07:15:37 +00:00
Your Name
f4998b3eee
fix(test): 修 P3.4 governance_agent 加第 5 項 slo_compliance 後既有測試對齊
...
CD Pipeline / build-and-deploy (push) Successful in 10m35s
P3.4 加入 check_slo_compliance 後：
- test_governance_agent::test_all_checks_fail_returns_all_errors: 4→5
- test_wave8_remaining_blockers::TestB8GovernanceFailureAlert: 三測試補 mock
🤖 Generated with [Claude Code](https://claude.com/claude-code )
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 15:06:58 +08:00
Your Name
8d6e086254
fix(p3.2): model_version_tracker 改 pure unit test + probe 改善
...
CD Pipeline / build-and-deploy (push) Failing after 2m7s
Engineer 重寫 test_model_version_tracker：
- 用 _make_fake_ctx (asynccontextmanager) 完整 mock get_db_context
- 移除 @pytest.mark.integration（整 class）
- patch probe_all_providers + get_db_context 雙路徑
- 4 testcases 全綠，無真實 PG 依賴
model_version_probe.py 配套改善（match 新 test mock 預期）
Tests: 19 passed (probe 15 + tracker 4)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 14:58:46 +08:00
Your Name
ed205489c1
feat(p3.2-tests+ci-schema): model_version 測試 + CI test_schema 對齊 + Grafana SLO Dashboard
...
CD Pipeline / build-and-deploy (push) Failing after 1m20s
P3.2 配套測試 + CI 環境同步 + ADR-100 Grafana 視覺化：
CI test_schema 補齊（解 1162-1172 阻塞之延伸）:
- setup_test_schema.sql 加 ai_provider_version_history 表
- 對齊 production p3_2_provider_version_history.sql（已 K8s exec 上線）
新增測試 (636 行):
- test_model_version_probe.py (387) — Provider 探測單元測試
- test_model_version_tracker.py (249) — Tracker 整合測試
· 4 個 DB-dependent tests 標 @pytest.mark.integration
· 15 unit + 4 integration（unit step 跳過 integration class）
新增配套:
- ai-slo-dashboard.json (496 行) — Grafana 儀表板
· 對應 ADR-100 SLO 規則的 4 大面板：
自主修復成功率 / 飛輪閉環延遲 / 治理事件 / Provider 健康度
修改:
- governance_agent.py +122 行 — SLO 指標暴露 + retrieve metric 整合
Tests: 15 passed (probe + tracker unit), 4 deselected (integration class)
Production 部署狀態:
- p2_decision_fusion_columns.sql ✅ K8s exec 完成（commit c58bdd0c）
- p3_2_provider_version_history.sql ✅ K8s exec 完成（this commit）
- 兩個 production migration 都已上線，CI test_schema 同步補齊
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 14:57:16 +08:00
Your Name
025a493f06
feat(p3.2+adr-100): Model Version Tracker + SLO 自治 + KB rot cleaner
...
run-migration / migrate (push) Failing after 12s
CD Pipeline / build-and-deploy (push) Has been cancelled
Wave 8 P3.2 模型版本追蹤 + ADR-100 SLO 自我治理 + 配套：
P3.2 — Model Version Tracking:
- model_version_probe.py (268 行) — 探測 Ollama / OpenRouter 等 provider 的 model version
- model_version_tracker.py (101 行) — 對齊 PG provider_version_history 表
- migrations/p3_2_provider_version_history.sql + rollback — 25 行 schema
- db/models.py +32 行 — ProviderVersionHistory ORM
ADR-100 — AI 自主化 SLO:
- docs/adr/ADR-100-ai-autonomous-slo.md (167 行) — 飛輪 SLO 設計與閾值
- ops/monitoring/slo-rules.yml (254 行) — Prometheus SLO recording rules + alerts
- ops/monitoring/tests/test_slo_rules.yaml (242 行) — promtool unit tests
整合修改:
- main.py +72 行 — Lifespan 啟動 model_version_probe + KB rot cleaner schedule
- gitea_webhook.py +45 行 — webhook 接收 model 版本變化通知
- ci_auto_repair.py / evidence_snapshot.py / pre_decision_investigator.py — 配合接線
新測試:
- test_kb_rot_cleaner_schedule.py (120 行) — 9 tests pass
- test_slo_rules.yaml — promtool 驗收
Tests: 9 passed (test_kb_rot_cleaner_schedule)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Multiple Engineers (P3.2 + ADR-100) <noreply@anthropic.com >
2026-04-27 14:54:19 +08:00
Your Name
9908fdf50d
feat(p3.1-t2-patha): DiagnosisAggregator 路徑 A + Solver F4 critical reject + 對齊測試
...
CD Pipeline / build-and-deploy (push) Failing after 1m59s
Wave 8 P3.1-T2 PathA 啟用 + Solver F4 安全強化 + test 對齊：
PathA — DiagnosisAggregator 信號分類層補 PDI:
- ENABLE_DIAGNOSIS_AGGREGATOR default=False → True
· PathA 純信號分類層（OOMKilled/CrashLoop 等業務邏輯）
· 不重複呼叫 K8s/SignOz API（只取 PDI 已收集的 raw 資料）
· 安全 default on — 純邏輯處理，無外部依賴重疊
- diagnosis_aggregator.py +155 行（PathA 實作）
- pre_decision_investigator.py 已接 (commit 3a2cd151noreply@anthropic.com >
Co-Authored-By: Multiple Engineers (Wave 8 P3.1-T2 PathA + F4) <noreply@anthropic.com >
2026-04-27 14:42:29 +08:00
Your Name
f09a8f56a9
fix(ci): test_schema 加 P2.1 fusion 欄位 — 解 CI 1162-1172 阻塞
...
CD Pipeline / build-and-deploy (push) Has been cancelled
Production PG migration 已上線（commit c58bdd0c），但 CI 用獨立 docker pgvector
test container（pg-test-b5），由 setup_test_schema.sql 初始化 → 無 fusion 欄位
→ test_b5_core_flows.py 整合測試失敗於 composite_score column does not exist。
修法：把 P2.1 ALTER TABLE 加入 setup_test_schema.sql（idempotent IF NOT EXISTS）
新增（對齊 production p2_decision_fusion_columns.sql）：
- composite_score REAL
- complexity_tier VARCHAR(16) + CHECK ('low','medium','high','critical')
- decision_fusion_details JSONB
partial index 不需要在 test schema（B5 整合測試不依賴 index）。
DO $$ block 處理 CHECK constraint 因 PG 不支援 ADD CONSTRAINT IF NOT EXISTS。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 14:39:06 +08:00
Your Name
fb130c9a28
feat(p3.1-t2): DiagnosisAggregator stub tests + sanitization 補強 + metrics 補欄
...
CD Pipeline / build-and-deploy (push) Failing after 2m16s
Wave 8 P3.1-T2 後續補測 + 配套：
新增測試:
- test_diagnosis_aggregator_stub.py (238 行) — 15 tests
· stub fixture 驗證 _collect_diagnosis_aggregator 接線
· feature flag default off 不呼叫
· timeout 邊界 / exception fail-soft
修改:
- core/metrics.py +23 — 新增 DiagnosisAggregator 相關 Prometheus 指標
- sanitization_service.py +24 — 補強 prompt sanitize 邊界（vuln #4 配套）
- RUNBOOK-AGENT-STEP-LATENCY.md / agent_step_latency_rules.yaml — 微調
Tests: 15 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:30:26 +08:00
Your Name
c58bdd0c38
chore(cd-trigger): production PG migration p2_decision_fusion_columns 已執行
...
統帥授權執行於 192.168.0.188:5432/awoooi_prod via K8s pod exec：
- composite_score REAL
- complexity_tier VARCHAR(16) + CHECK ('low','medium','high','critical')
- decision_fusion_details JSONB
- ix_approval_composite_score (partial, WHERE composite_score IS NOT NULL)
- ix_approval_complexity_tier (partial, WHERE complexity_tier IS NOT NULL)
Pre-existing CI integration test 阻塞解，全部 25+ commits 應一次部署。
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:29:57 +08:00
Your Name
9a711278f7
test(p3.1-t2): Sentry Webhook 簽章驗證 dedicated tests
...
CD Pipeline / build-and-deploy (push) Failing after 1m23s
對應 commit 3a2cd151noreply@anthropic.com >
2026-04-27 08:24:59 +08:00
Your Name
2b39558492
test(governance): trust_drift_watchdog dedicated tests
...
CD Pipeline / build-and-deploy (push) Has been cancelled
P2.2 governance 補測：trust_drift watchdog 9 個整合測試。
Tests: 9 passed
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:24:37 +08:00
Your Name
3a2cd15144
feat(p3.1-t2): Tier-2 三服務感知強化 — Sentry 簽章 + DiagnosisAggregator + Solver actions test
...
CD Pipeline / build-and-deploy (push) Has been cancelled
Wave 8 P3.1-T2 三項感知強化（多 engineer 補完）：
Sentry Webhook 簽章驗證:
- sentry_webhook.py: 接入 SentryWebhookService.verify_sentry_signature()
- 拒絕無效 sentry-hook-signature → 401 → 防偽造攻擊
DiagnosisAggregator Pod 深診斷整合:
- pre_decision_investigator.py: 新增 _collect_diagnosis_aggregator()
- ENABLE_DIAGNOSIS_AGGREGATOR feature flag 守衛（default=False）
- evidence_snapshot.py: extra_diagnosis 欄位 + build_summary 顯示
- timeout=3.0s + try/except 隔離（fail-soft）
- Conservative 策略：待重疊分析確認 vs PreDecisionInvestigator 不重複
config.py:
- 新增 ENABLE_DIAGNOSIS_AGGREGATOR Field（default=False，K8s ConfigMap 動態啟用）
Solver B1 補測（commit 7c726ebcnoreply@anthropic.com >
Co-Authored-By: Multiple Engineers (Wave 8 P3.1-T2) <noreply@anthropic.com >
2026-04-27 08:24:15 +08:00
Your Name
6de10cb073
test(wave8-blockers): 4 餘項 BLOCKER 修復驗收（vuln #4 + B14 + B25/B26 + B8）
...
CD Pipeline / build-and-deploy (push) Has been cancelled
確認 critic + debugger + vuln-verifier 報告中尚未驗收的 4 修復都已實裝在 production，
並補對應 dedicated tests:
vuln #4 — fusion prompt injection 防禦:
- score_with_elephant 內 _sanitize 剔除控制字元 + 截長至 max_len
- alert_name(100) / evidence(...) / proposal(300) 三層 sanitize
- 驗證：1000 個 'A' 攻擊 payload → prompt 內 'A' < 200，控制字元 \\x00\\x1b\\x02 全剔除
debugger B14 — Gemini quota fail-closed:
- ollama_failover_manager._check_gemini_quota except branch
- Redis 異常時 return False（非 fail-open），費用安全 > 服務可用性
- best-effort 呼叫 alert_gemini_quota_exceeded 通知運維
debugger B25/B26 — auto_repair drain_pending_tasks:
- AutoRepairService._pending_tasks (set) + drain_pending_tasks(timeout=60.0)
- main.py shutdown 已接 _repair_svc.drain_pending_tasks() 呼叫
- K8s rolling restart 時 fire-and-forget tasks 不丟失
debugger B8 — governance ≥3 failures alert:
- run_self_check 後聚合 failed_checks
- ≥3 項失敗 → self._alert("governance_self_failure", ...) 觸發
- payload 含 failed_checks list + total_checks=4 + errors dict
Tests: 10/10 PASSED (vuln 3 + B14 2 + drain 2 + governance 3)
Note: 此 commit 純補測，所有 4 修復代碼上 commit 已 in production
仍待: 1167+ CD runs 確認 deploy 成功
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-27 08:22:47 +08:00
Your Name
7c726ebc1c
fix(b1): Solver Agent 結構化動作 — 北極星 §1.1 修復多樣性 ≥ 40%
...
CD Pipeline / build-and-deploy (push) Failing after 2m22s
INC-20260425 衍生修復 — Solver 拒絕 rule-based mock 兜底:
原設計缺陷:
- LLM 失敗時 → rule-based mock 推 RESTART 兜底
- 違反北極星 §1.1：修復多樣性 ≥ 40%（不能寫死同一指令）
新設計:
- LLM 失敗 → graceful degraded（candidates=[], recommended_actions=[], degraded=True）
- 禁止 rule-based mock / hardcode RESTART
- 新增 recommended_actions 結構化 MCP 動作清單
· 供 B3 Telegram 按鈕動態生成
· YAML 規則庫驅動，非寫死
- 新增 yaml + Path import 載入動作模板庫
向下相容:
- 既有 candidates / blast_radius 邏輯不變
- 新增欄位 recommended_actions 為 optional list
Tests: 8 passed (solver 相關全綠)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Claude Sonnet 4.6 (B1 北極星 §1.1) <noreply@anthropic.com >
2026-04-27 08:18:38 +08:00
Your Name
21977004e7
test(p3.1-t1): test_p3_tier1_integrations 對應 model_rollback + resource_resolver 整合
...
CD Pipeline / build-and-deploy (push) Has been cancelled
P3.1-T1 接線測試（補 commit 123d9c8anoreply@anthropic.com >
2026-04-27 08:17:59 +08:00
Your Name
123d9c8a2e
fix(p3.1-t1): 三 Tier-1 服務整合 — model_rollback_service + resource_resolver
...
CD Pipeline / build-and-deploy (push) Has been cancelled
P3.1-T1 接線兩個既有服務到主流程：
offline_replay_service.py — model_rollback_service 整合:
- 回放事件寫入治理 DB 後，觸發 ModelRollbackService.check() 衰退偵測
- feature flag 由 model_rollback_service 自行判斷（AIOPS_P6_GOVERNANCE_ENABLED）
- retrain_recommended → log warning 含 streak / absolute_floor / conservative_mode
- exception fail-soft（不阻斷 replay 主流程）
approval_execution.py — resource_resolver 整合:
- kubectl 指令解析後，動態驗證資源是否存在於 K8s
- 若 resolved_name != raw_name → log + apply normalized name
- 若不存在但有 candidates → log warning + suggestions（不攔截執行，只記錄）
- exception fail-soft（不阻斷主流程）
- RESOURCE_RESOLVE_TOTAL Prometheus counter labels: hit/suggestion/miss/error
Tests: 後端 1303 collected（無回歸），對應 dedicated 測試在前次 commit 已寫
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Claude Sonnet 4.6 (P3.1-T1) <noreply@anthropic.com >
2026-04-27 08:17:04 +08:00
Your Name
fefe4c21cd
fix(inc-20260425): A1+A2 後續 — Solver/Critic timeout + auto_repair 接線 + Runbook + Grafana
...
CD Pipeline / build-and-deploy (push) Has been cancelled
延續 595629c0noreply@anthropic.com >
Co-Authored-By: Claude Sonnet 4.6 (INC-20260425 後續) <noreply@anthropic.com >
2026-04-27 08:15:53 +08:00
Your Name
595629c013
fix(inc-20260425): A1 三段 Agent timeout 拆分 + A2 DIAGNOSE 移除 Ollama
...
CD Pipeline / build-and-deploy (push) Has been cancelled
INC-20260425-8D17BB / 3B6C39 兩則告警 AI 信心降到 20% 根因雙修（統帥批准 A+B）：
A1 — 三段 Agent step timeout 拆分（北極星 §1.2 Observable by Default）:
- diagnostician_agent.py: PHASE2_STEP_TIMEOUT_SEC=20.0 共用值 → 拆三段
· AGENT_DIAGNOSTICIAN_TIMEOUT_SEC=30.0（NIM 主吃口，最大 prompt + 多假設）
· AGENT_SOLVER_TIMEOUT_SEC=20.0（後續 commit 接線）
· AGENT_CRITIC_TIMEOUT_SEC=15.0（後續 commit 接線）
· env override 支援，K8s ConfigMap 動態調整不需 rebuild
· 保留 PHASE2_STEP_TIMEOUT_SEC alias（DEPRECATED，下 sprint 移除）
- observability/agent_step_metrics.py (58 行) — 新模組:
· aiops_agent_step_duration_seconds Histogram
· observe_agent_step() helper 統一三 Agent 呼叫點
· outcome label ∈ {success, timeout, error}
· agent label ∈ {diagnostician, solver, critic}
A2 — ai_router DIAGNOSE chain 移除 Ollama:
- ai_router.py v4.4 by Claude Sonnet 4.6
· 新增 _diagnose_fallback_chain: NEMO → GEMINI → CLAUDE
· Ollama 永久排除於此 chain（CPU-only 實測 238s，二次 timeout 必爆）
· 新增 aiops_diagnose_fallback_total Prometheus metric
- 根因: NIM timeout 後 fallback 到 Ollama deepseek-r1:14b CPU 238s
→ 二次 timeout → degraded confidence=0.2
Wave8-X2 整合測試補正:
- test_ollama_failover_manager.py: TestSelectProvider 補 mock _check_gemini_quota
原 test 期望 OFFLINE→Gemini，但 quota fail-closed 後沒 mock 會被切到 188
繞過 quota check 後驗純路由邏輯 → 37/37 PASS
Tests: 37 passed (test_ollama_failover_manager 全部)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Claude Sonnet 4.6 (Wave 8 INC-20260425) <noreply@anthropic.com >
2026-04-27 08:15:10 +08:00
Your Name
1ab6786ce3
feat(ops): Ollama 容災 Runbook + Grafana 儀表板 + Consensus K8s ConfigMap patch
...
run-migration / migrate (push) Failing after 13s
CD Pipeline / build-and-deploy (push) Failing after 2m1s
Wave 6 P2.3 ops 配套 + tool-expert 部署文件：
新增:
- docs/runbooks/RUNBOOK-OLLAMA-FAILOVER.md (240 行)
· 三大鐵律驗證步驟（自動切 Gemini / 自動切回 / quota 熔斷）
· failover/recovery 完整 SOP
· 故障排查清單（Ollama 111/188 不通、Gemini quota 超發等）
- ops/monitoring/grafana/dashboards/ollama_failover.json (295 行)
· 4 panel：current primary / failover events / quota usage / health status
· 對應 P2.3 metrics: OLLAMA_FAILOVER_TRIGGERED_TOTAL / GEMINI_DAILY_CALL_COUNT
- k8s/awoooi-prod/04-configmap.yaml.patch-consensus
· ENABLE_12AGENT_CONSENSUS / ENABLE_AIOPS_P2_FUSION feature flag patch
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: tool-expert agent (Wave 6) <noreply@anthropic.com >
2026-04-27 08:11:40 +08:00
Your Name
1096da12ae
feat(p2.5): aiops 時序前端面板 — Incident 6 階段視覺化
...
Wave 6 P2.5 frontend-designer 工業級視覺化（拒絕 AI slop）：
新增（1824 行）:
- apps/web/src/app/[locale]/aiops/timeline/page.tsx
- apps/web/src/components/aiops/timeline/
· AiopsTimelinePanel.tsx (413) — 主面板組件
· TimelineStage.tsx (279) — 6 階段時序卡片
· TimelineStageDetails.tsx (359) — 階段細節展開
· EvidenceViewer.tsx (144) — Evidence Snapshot 檢視
· TimelineFilter.tsx (109) — incident_id / severity / 時段 過濾器
· types.ts (118) — TS 型別定義
· mock-data.ts (357) — 開發 mock fallback
· index.ts (7) — barrel export
- i18n: messages/en.json + messages/zh-TW.json — Timeline 翻譯
設計原則:
- 拒絕 AI slop（無泛用 emoji/漸層，採工業 dashboard 風格）
- 後端 endpoint 接通 /api/v1/aiops/timeline（critic B4 修復）
- mock 模式 fallback 防 endpoint 暫時不可達
對應後端: a3b4595e（aiops_timeline.py + aiops_timeline_service.py）
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: frontend-designer agent (Wave 6) <noreply@anthropic.com >
2026-04-27 08:11:40 +08:00
Your Name
cc547736ab
feat(wave6-8): P2.1 fusion + P2.2 governance + P2.4 consensus + Wave 7/8 BLOCKER 修復
...
承接 Wave 6/7/8 多 engineer 在 agent 限額前完成的代碼，補 commit 解 production
HEAD 隱性 import error（decision_fusion 已被 decision_manager 引用但檔案 untracked）。
新增（後端核心）:
- decision_fusion.py (562 行) — P2.1 方法 III（OpenClaw + Hermes + Elephant 三 LLM 融合）
- aiops_timeline.py + aiops_timeline_service.py — critic B4 修復
/api/v1/aiops/timeline endpoint，DB 存取抽到 service 層遵守 leWOOOgo 積木化
- migrations/p2_decision_fusion_columns.sql + rollback — approval_records fusion 欄位
修改（後端整合）:
- decision_manager.py — fusion 三斷鏈修補（critic B1+B2+B3）：
· B1: 寫 _evidence_snapshot_ref 到 token.proposal_data
· B2: fusion 前計算 complexity_score 並寫 token
· B3: fusion composite 寫 token.proposal_data["decision_fusion"]
- auto_approve.py — fusion + consensus 認識（critic B3+B5）：
· composite > 0.7 → auto_execute_eligible bypass min_confidence
· source=consensus_engine + score>=0.6 → 規則可信路徑
- consensus_engine.py — db-fix _save_consensus 重用 agent_sessions
- governance_agent.py — db-fix _alert PG 寫入 ai_governance_events
- approval_db.py — fusion 3 欄位 + 2 partial index + CheckConstraint
- db/models.py — schema 對齊 migration
- core/config.py — vuln #1 修復：OLLAMA_URL/_FALLBACK_URL field_validator
拒絕公網 IP + 外部域名，僅允許私網/loopback/K8s SVC 白名單
- core/feature_flags.py — P2 fusion + consensus flags
- main.py — governance_agent lifespan 啟動
- failover_alerter.py — Wave8-X2: in-memory dedup fallback（Redis 拒絕後不 fail-open）
- ollama_*.py — metrics 整合 + recovery 改善
- auto_repair_service.py — verifier 接線
新增（測試 2438 行）:
- test_decision_fusion.py / test_governance_agent.py / test_consensus_integration.py
- test_p2_db_fixes.py / test_wave8_fusion_fixes.py
- test_config_url_validation.py（vuln #1 12 tests）
- test_failover_alerter.py +Wave8-X2 in-memory dedup 補測
驗收: 116 tests pass (decision_fusion + wave8_fusion + config_url + consensus +
governance + p2_db_fixes + failover_alerter)
Conflict resolution:
- 3 檔（config.py + auto_approve.py + decision_manager.py）git stash pop 衝突
保留 stashed (engineer 最終版)，補回 ValueError 「公網 IP」字樣對齊 test
Note: 此 commit 解 production HEAD 隱性 import error
仍未修: vuln #4 prompt injection / debugger B14 quota fail-closed
/ B25-B26 drain_pending_tasks / B8 governance fail alert
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
Co-Authored-By: Multiple Engineers (Wave 6/7/8) <noreply@anthropic.com >
2026-04-27 08:11:40 +08:00
AWOOOI CD
b0bf3783e4
chore(cd): deploy 2c57b71 [skip ci]
2026-04-26 13:04:37 +00:00
