Commit Graph

1946 Commits

Author SHA1 Message Date
Your Name
fa6a78af2a chore(cd): deploy e4aef6a api [skip ci] 2026-05-01 10:42:07 +08:00
Your Name
e4aef6ac4e fix(aiops): block k8s playbooks for host repair
All checks were successful
CD Pipeline / tests (push) Successful in 1m27s
Code Review / ai-code-review (push) Successful in 26s
CD Pipeline / build-and-deploy (push) Successful in 8m6s
CD Pipeline / post-deploy-checks (push) Successful in 3m31s
2026-05-01 10:33:52 +08:00
AWOOOI CD
7472eb2fcd chore(cd): deploy ca22ec2 [skip ci] 2026-05-01 10:24:48 +08:00
Your Name
ca22ec2fd2 fix(aiops): route backup failures rule-first
All checks were successful
CD Pipeline / tests (push) Successful in 1m51s
Code Review / ai-code-review (push) Successful in 30s
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 42s
CD Pipeline / build-and-deploy (push) Successful in 8m21s
CD Pipeline / post-deploy-checks (push) Successful in 4m18s
2026-05-01 10:11:10 +08:00
AWOOOI CD
3e0ab0f8c6 chore(cd): deploy f154ac0 [skip ci] 2026-05-01 00:14:36 +08:00
Your Name
f154ac022e feat(playbook): version generated playbooks
All checks were successful
CD Pipeline / tests (push) Successful in 1m34s
Code Review / ai-code-review (push) Successful in 28s
Type Sync Check / check-type-sync (push) Successful in 1m10s
CD Pipeline / build-and-deploy (push) Successful in 10m19s
CD Pipeline / post-deploy-checks (push) Successful in 3m1s
2026-04-30 23:59:39 +08:00
Your Name
474b913ac9 chore(db): add playbook versioning migration
Some checks failed
CD Pipeline / tests (push) Successful in 1m32s
Code Review / ai-code-review (push) Successful in 27s
run-migration / migrate (push) Failing after 13s
CD Pipeline / build-and-deploy (push) Has started running
CD Pipeline / post-deploy-checks (push) Has been cancelled
E2E Health Check / e2e-health (push) Successful in 43s
2026-04-30 23:53:19 +08:00
Your Name
f0d14ab6c4 fix(aiops): escalate blocked auto repair
Some checks failed
CD Pipeline / tests (push) Successful in 1m33s
Code Review / ai-code-review (push) Successful in 28s
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 40s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled
2026-04-30 23:49:17 +08:00
AWOOOI CD
f946e7b184 chore(cd): deploy 6e04fe9 [skip ci] 2026-04-30 23:18:20 +08:00
Your Name
7d02365dc2 chore(types): sync playbook enums
All checks were successful
Type Sync Check / check-type-sync (push) Successful in 1m14s
2026-04-30 23:10:37 +08:00
Your Name
6e04fe9c8a feat(playbook): generate drafts with local llm
Some checks failed
CD Pipeline / tests (push) Successful in 1m28s
Code Review / ai-code-review (push) Successful in 29s
Type Sync Check / check-type-sync (push) Failing after 2m41s
CD Pipeline / build-and-deploy (push) Successful in 8m40s
CD Pipeline / post-deploy-checks (push) Successful in 3m10s
2026-04-30 23:04:58 +08:00
Your Name
95110971f3 fix(telegram): close remaining DM alert routes
Some checks failed
CD Pipeline / tests (push) Successful in 1m27s
Code Review / ai-code-review (push) Successful in 29s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled
2026-04-30 23:02:17 +08:00
AWOOOI CD
64b09273f7 chore(cd): deploy e29aab5 [skip ci] 2026-04-30 15:58:18 +08:00
Your Name
e29aab5a52 fix(cd): write smoke output in workspace
All checks were successful
CD Pipeline / tests (push) Successful in 1m28s
Code Review / ai-code-review (push) Successful in 25s
CD Pipeline / build-and-deploy (push) Successful in 6m56s
CD Pipeline / post-deploy-checks (push) Successful in 3m6s
2026-04-30 15:49:33 +08:00
AWOOOI CD
a93fbe5d66 chore(cd): deploy 36967d0 [skip ci] 2026-04-30 15:44:46 +08:00
Your Name
36967d04ac fix(cd): allow smoke status output writes
All checks were successful
CD Pipeline / tests (push) Successful in 1m22s
Code Review / ai-code-review (push) Successful in 26s
CD Pipeline / build-and-deploy (push) Successful in 6m50s
CD Pipeline / post-deploy-checks (push) Successful in 2m54s
2026-04-30 15:36:11 +08:00
AWOOOI CD
38ffcf4395 chore(cd): deploy 712d3e5 [skip ci] 2026-04-30 15:20:33 +08:00
AWOOOI CD
ae52d51210 chore(cd): deploy 72945bf [skip ci] 2026-04-30 15:05:57 +08:00
Your Name
712d3e5a77 fix(ci): send workflow alerts to SRE group
All checks were successful
CD Pipeline / tests (push) Successful in 1m30s
Code Review / ai-code-review (push) Successful in 26s
CD Pipeline / build-and-deploy (push) Successful in 7m48s
CD Pipeline / post-deploy-checks (push) Successful in 2m58s
2026-04-30 15:05:16 +08:00
Your Name
61f5a6a419 fix(telegram): route alerts to SRE war room
Some checks failed
CD Pipeline / tests (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled
CD Pipeline / post-deploy-checks (push) Has been cancelled
Code Review / ai-code-review (push) Has been cancelled
2026-04-30 15:01:23 +08:00
Your Name
72945bf283 chore(cd): retry post deploy after runner restore 2026-04-30 14:48:28 +08:00
AWOOOI CD
6e76c5dfd5 chore(cd): deploy c9393c3 [skip ci] 2026-04-30 14:41:46 +08:00
Your Name
c9393c3688 fix(cd): run post deploy checks on host runner
Some checks failed
Code Review / ai-code-review (push) Successful in 27s
CD Pipeline / tests (push) Successful in 2m46s
CD Pipeline / build-and-deploy (push) Successful in 7m46s
CD Pipeline / post-deploy-checks (push) Failing after 19s
2026-04-30 14:31:12 +08:00
AWOOOI CD
19788302df chore(cd): deploy 80defbe [skip ci] 2026-04-30 14:26:44 +08:00
Your Name
80defbed7c fix(aiops): fallback and escalate automation blockers
Some checks failed
CD Pipeline / tests (push) Successful in 2m41s
Code Review / ai-code-review (push) Successful in 24s
CD Pipeline / build-and-deploy (push) Successful in 7m51s
CD Pipeline / post-deploy-checks (push) Failing after 2m15s
2026-04-30 14:13:57 +08:00
Your Name
82649c2cbb fix(cd): run tests in explicit ci container
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled
CD Pipeline / tests (push) Has been cancelled
CD Pipeline / post-deploy-checks (push) Has been cancelled
Code Review / ai-code-review (push) Has been cancelled
2026-04-30 14:11:39 +08:00
Your Name
ed2a4838f2 fix(auto): use action parser for repair gates
Some checks failed
CD Pipeline / tests (push) Failing after 1m2s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped
Code Review / ai-code-review (push) Successful in 24s
2026-04-30 14:06:09 +08:00
AWOOOI CD
9ee3cc6242 chore(cd): deploy 4723499 [skip ci] 2026-04-30 11:11:04 +08:00
Your Name
4723499955 fix(cd): install playwright system deps for smoke
All checks were successful
CD Pipeline / tests (push) Successful in 1m34s
Code Review / ai-code-review (push) Successful in 24s
CD Pipeline / build-and-deploy (push) Successful in 6m58s
CD Pipeline / post-deploy-checks (push) Successful in 3m7s
2026-04-30 11:02:12 +08:00
Your Name
e27b462bef fix(ops): keep disabled gitea runner stopped
All checks were successful
Code Review / ai-code-review (push) Successful in 27s
2026-04-30 10:59:46 +08:00
AWOOOI CD
a0be4ebb03 chore(cd): deploy 0f7e9d3 [skip ci] 2026-04-30 10:54:29 +08:00
Your Name
0f7e9d3467 fix(cd): run docker builds on host runner
All checks were successful
CD Pipeline / tests (push) Successful in 1m33s
Code Review / ai-code-review (push) Successful in 25s
CD Pipeline / build-and-deploy (push) Successful in 9m20s
CD Pipeline / post-deploy-checks (push) Successful in 1m33s
2026-04-30 10:43:33 +08:00
Your Name
7cc10b2599 fix(cd): serialize gitea docker builds
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 40s
Code Review / ai-code-review (push) Successful in 24s
2026-04-30 10:11:50 +08:00
Your Name
e91db52858 docs(logbook): record 639bb64 prod deployment [skip ci] 2026-04-30 09:45:48 +08:00
Your Name
9f15f3cfe4 chore(cd): deploy 639bb64 [skip ci] 2026-04-30 09:41:20 +08:00
Your Name
639bb64788 feat(flywheel): surface ai automation and code review
Some checks failed
Code Review / ai-code-review (push) Successful in 31s
CD Pipeline / build-and-deploy (push) Failing after 5m23s
2026-04-30 00:09:25 +08:00
AWOOOI CD
d197e2785d chore(cd): deploy 4a57c2d [skip ci] 2026-04-29 15:48:24 +00:00
Your Name
4a57c2d04f feat(flywheel): expose incident processing timeline
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 10m56s
2026-04-29 23:38:30 +08:00
AWOOOI CD
dae0aa2312 chore(cd): deploy d845d53 [skip ci] 2026-04-29 15:06:57 +00:00
Your Name
d845d53257 fix(security): keep Gemini key out of request URLs
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 15m5s
2026-04-29 22:56:12 +08:00
AWOOOI CD
b857be0a64 chore(cd): deploy fe2b8f4 [skip ci] 2026-04-29 14:47:51 +00:00
Your Name
fe2b8f4571 fix(flywheel): fallback on OpenClaw degraded responses
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 9m56s
2026-04-29 22:38:57 +08:00
AWOOOI CD
525a243550 chore(cd): deploy dccdcdb [skip ci] 2026-04-29 13:59:53 +00:00
Your Name
dccdcdbaf5 fix(flywheel): unblock action safety and Claude fallback
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 9m45s
2026-04-29 21:51:18 +08:00
AWOOOI CD
4c91d89dd2 chore(cd): deploy 4115ddd [skip ci] 2026-04-29 13:04:37 +00:00
Your Name
f5f41543c9 docs: ADR-105 推翻 A2 + LOGBOOK 2026-04-29 LLM 飛輪復活戰
ADR-105 完整記錄推翻 A2 鐵律的決策:
- Context: A2 歷史背景 + 2 個月後事實基礎變化(GPU + qwen2.5:7b)
- Decision: 4 處修改(IntentType.DIAGNOSE override / chain / openclaw.py task_type / 6 regression test)
- Consequences: 正面(飛輪復活)+ 負面(Ollama 單點)+ 已知債(ADR-106-109 後續)
- Validation: 部署前 1635 tests 全綠,部署後 5 項驗證指標
- Rollback: env 切換 / git revert

LOGBOOK 加 2026-04-29 條目:
- 真根因:4 provider 全死 + A2 鐵律排除 Ollama
- CD 連環血淚:5 個 commit 全 failure(setup_test_schema.sql 缺欄)
- 已落地(不依賴 CD):Prometheus 17 條 rule + Gemini sanitize
- Memory 索引同步更新(指向 project_revert_a2_ollama_primary.md)

注意:docs/ 不在 cd.yaml paths trigger,此 commit 不影響 CD。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 20:59:53 +08:00
Your Name
4115ddde48 fix(cd-blocker-2): setup_test_schema.sql 補 KM 欄位(解 CD 真實 root cause)
All checks were successful
CD Pipeline / build-and-deploy (push) Successful in 14m4s
## 之前 c5b18101 修錯地方

我加 db/base.py:init_db() ALTER 沒解問題。**CI 不跑 init_db()**。

## 真實 CD 流程

`.gitea/workflows/cd.yaml` Integration Tests step:
1. 啟動臨時 `pg-test-b5` 容器(fresh PG)
2. `psql -f tests/integration/setup_test_schema.sql` 建表
3. 跑 pytest tests/integration/test_b5_core_flows.py

setup_test_schema.sql 的 `knowledge_entries` 表沒有
`related_approval_id` + `path_type` 欄位 → INSERT 失敗。

## 修法

setup_test_schema.sql:110 `CREATE TABLE knowledge_entries` 補:
- related_approval_id VARCHAR(64)
- path_type VARCHAR(50)
- uix_knowledge_incident_path partial unique index
- ix_knowledge_related_approval partial index

## 預期效果

CD #1119 (本 commit) 應該成功。
解鎖 4 個 stuck commit (1114-1118) 的部署 backlog。
fb0c72db 推翻 A2 DIAGNOSE Ollama primary 終於上 prod。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 20:54:54 +08:00
Your Name
c5b1810172 fix(cd-blocker): 補 knowledge_entries 防禦性 ALTER(解 CD #1115-1117 全 failure)
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m38s
🚨 真根因:CD pipeline 從昨天 push fb0c72db 起,4 個 commit (1114-1117) 全 failure
prod pod 28 小時沒更新 → 統帥 17:33/17:35 看到的 Telegram 告警仍是「llm_failed」
不是 ai_router 沒推翻 A2,是**部署根本沒上 prod**。

## CD 失敗證據(gitea actions API)

```
#1117 7b471e7a failure  Gemini sanitize
#1116 3668d49f failure  W2 三件 + KMWriter critic
#1115 fb0c72db failure  推翻 A2 DIAGNOSE Ollama primary
#1114 8d24f151 failure  PR-R1 4 Major 修
#1113 681b5ac9 success  PR-R1 規則→Playbook 遷移  ← 最後一次成功
```

## 失敗 Stack Trace(job 1267 logs)

```
sqlalchemy.exc.ProgrammingError: column "related_approval_id"
of relation "knowledge_entries" does not exist
SQL: INSERT INTO knowledge_entries (..., related_approval_id, path_type, ...)
test: tests/integration/test_b5_core_flows.py::test_knowledge_entry_view_count
```

## 根因

commit c22e5f33 (KMWriter) 加 ORM 欄位 `related_approval_id` + `path_type`:
- `models.py` ORM Mapped 欄位 
- `knowledge.py` Pydantic schema 
- `migrations/p1_1_km_idempotent_path_type.sql` 加 path_type 
- **但 `db/base.py:init_db()` 沒對應 ALTER**

CI integration test 用 prod schema 建 PG → 既有表沒有新欄位 → INSERT 失敗。
我之前只補了 `timeline_events.incident_id` 的 ALTER,漏了 `knowledge_entries`。

## 修法

`db/base.py:init_db()` 補 3 條防禦性 SQL(同 timeline_events 模式):
```sql
ALTER TABLE knowledge_entries
    ADD COLUMN IF NOT EXISTS related_approval_id VARCHAR(64),
    ADD COLUMN IF NOT EXISTS path_type VARCHAR(50);
CREATE UNIQUE INDEX IF NOT EXISTS uix_knowledge_incident_path
    ON knowledge_entries(related_incident_id, path_type)
    WHERE related_incident_id IS NOT NULL AND path_type IS NOT NULL;
CREATE INDEX IF NOT EXISTS ix_knowledge_related_approval
    ON knowledge_entries(related_approval_id)
    WHERE related_approval_id IS NOT NULL;
```

## 驗證

- 1635 unit tests 全綠
- 預期 CD #1118 (本 commit) 解 4 個失敗 commit 的部署 backlog
- 部署完成後 prod ai_router fb0c72db 推翻 A2 才會真的生效

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 20:44:23 +08:00
Your Name
7b471e7ae2 fix(secret-leak): Gemini API key 從 prod log 清除(P0 SECRET LEAK)
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 2m6s
## 問題(2026-04-29 11:50 prod log 證據)

prod log 出現完整 Gemini API key 明碼:
```
"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=AIzaSyCqv7TY2iTGi2wa91d2irwH08VYXjT9YUk"
event: gemini_provider_failed
```

違反鐵律:
- feedback_secret_debug_output_ban.md: debug 含 secret 字串禁 echo/log 原值
- feedback_secrets_leak_incidents_2026-04-18.md: 已有 2 起 secret leak 事故

## 根因

`gemini.py:118` `logger.warning("gemini_provider_failed", error=str(e), ...)`

httpx HTTPStatusError str() 會包含完整 URL(含 ?key=... query string):
- Google Gemini API 設計用 query string 傳 API key(不像 Claude/NVIDIA 用 header)
- httpx 拋例外時把 URL 寫進 error message
- str(e) 直接 log → key 進 K8s pod log → audit log → Sentry → 任何下游 log 接收方

## 修法

新增 `_sanitize_error()` 函式:
- regex `([?&])key=[^&\s'"]+` → `\1key=<redacted>`
- 在 `gemini_provider_failed` log 出口呼叫
- AIResult.error 也用 sanitize 過的(不污染下游)

只修 Gemini(其他 provider 用 header / 內網無 key):
- Claude: API key 在 `x-api-key` header → 不在 URL → 安全
- OpenClaw: 內網 188:8088 → 無 API key → 安全
- Ollama: 內網 111:11434 → 無 API key → 安全
- NVIDIA: API key 在 `Authorization: Bearer` header → 安全

## 驗證

- 1635 unit tests 全綠(修法不破壞任何既有行為)
- 直接執行 sanitize 函式確認 `AIzaSy*` key 被替換成 `<redacted>`

## 已知債

- 此 commit 只防新 leak,**舊 log 中的 key 仍存在**(K8s pod log / Sentry / structlog backend)
- Gemini API key 仍應**輪換**(已洩漏的 key 不可信)
- 統帥需手動:
  1. 去 https://aistudio.google.com/apikey 新增 key
  2. 在 K8s secret 換 GEMINI_API_KEY
  3. 撤銷舊 key

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 19:49:09 +08:00
Your Name
3668d49f2f feat(flywheel): W2 三件 + KMWriter critic 修法(1635 tests 全綠)
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 1m38s
W2 (onboarder 4 週飛輪 80→90 路徑第二週) + critic PR review 5 個 critical/major
全部修完,default flag=false 安全無爆炸風險。

## W2 三件 PR

### PR-R2 — AOL → catalog confidence EWMA 回灌(修飛輪斷鏈 C2)
- 新檔 `apps/api/src/jobs/aol_to_catalog_writeback_job.py`
- 邏輯:每小時掃 AOL 計算 EWMA confidence (alpha=0.3) 回灌 alert_rule_catalog
- 失敗閾值 N=5 連續低成功率 → review_status='draft'
- Hermes _fetch_noisy_rules SQL 加 OR review_status='draft'
- ENABLE_AOL_WRITEBACK_JOB=false (default)
- 8 個測試(mock path 修正:lazy import → patch src.db.base.get_db_context)

### PR-V1 — self_healing_validator 串接 (修飛輪斷鏈 C6)
- 新檔 `apps/api/src/services/self_healing_validator.py`(純函數 assess_self_healing)
- post_execution_verifier.py step 5 串接(feature flag gate)
- evidence_snapshot.py 加 self_healing_score / self_healing_detail 欄位
- db/models.py + base.py ALTER IF NOT EXISTS
- score < 0.5 → 觸發 rollback 提案 Telegram alert(不自動執行)
- ENABLE_SELF_HEALING_VALIDATOR=false (default)
- 7 個測試

### PR-L1 — KM ↔ Playbook 雙向回路 (修飛輪斷鏈 C3+C4)
- learning_service.py 三條新邏輯:
  1. _write_playbook_evolution_km:promote/demote 寫 KM 演化條目
  2. _check_and_mark_playbook_review:N=5 累積觸發 review_required
  3. _demote_alert_rule_catalog_confidence:DEPRECATED → confidence×=0.5
- PlaybookRecord 加 review_required 欄位(schema migration via base.py)
- ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=false (default)
- KM_PLAYBOOK_REVIEW_THRESHOLD=5 可調
- 6 個測試

## KMWriter Critic 5 個 Critical/Major 修復(之前 critic PR review 發現)
之前 push commit c5753e1c 已修,本 commit 補回 stash 中的對應檔案:
- C1 km_writer.py:194 backfill 自打臉(已修:同步 await + DLQ)
- C2 km_writer.py:391 KM_WRITE_AWAIT=false 路徑收緊
- M1 decision_manager.py:2178/2203 移除 _fire_and_forget
- M2 incident_service.py:1099 自製 path 加 retry+DLQ
- M3 km_writer.py:166 冪等聲明對齊(UPSERT + partial unique index)

## 驗證
- 1635 unit tests 全綠(+27 from 1608)
- 與 fb0c72db (推翻 A2 Ollama primary) 共存無衝突
- 所有新 Job/Service default flag=false(不爆炸)

## 期望影響
飛輪斷鏈 C2 + C3 + C4 + C6 全修
飛輪自主化評分:65 → 85 預估(W2 完成後)

啟用順序(待 prod fb0c72db 驗證 OLLAMA primary 跑得起來後):
1. ENABLE_AOL_WRITEBACK_JOB=true
2. ENABLE_KM_PLAYBOOK_FEEDBACK_LOOP=true
3. ENABLE_SELF_HEALING_VALIDATOR=true

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-29 19:44:04 +08:00