wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

docs(monitoring): record t103 alert chain evidence rollout [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-05-20 13:29:37 +08:00
parent 4956fbb849
commit ef811c979b
2 changed files with 77 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md
View File

@@ -2402,6 +2402,14 @@ Phase 6 完成後
- Production / CI：`8fa8d690 fix(monitoring): stabilize post-deploy target coverage`、`6e5d68ee test(monitoring): avoid script bytecode cleanup noise`、`89f39759 ci: clean b5 test bytecode artifacts` 已推 Gitea main；deploy marker `f542aa52 chore(cd): deploy 6e5d68e [skip ci]`。Actions：#1911 Code Review success 10s、#1910 CD success 7m17s、#1913 Code Review success 10s、#1912 CD success 9m20s、#1914 Code Review success 12s。Post-CD smoke：health healthy/prod/mock_mode=false；Monitoring Coverage Check coverage 100.0%、real_down_jobs=0、`awoooi-api` UP；Playwright E2E 5 passed。
- 目前進度更新：AwoooP 告警可觀測鏈約 99.7%；Monitoring coverage gate 準確性約 99.7%；低風險自動修復閉環約 95.2%；前端 AI 自動化管理介面同步約 99.6%；完整 AI 自動化管理產品化約 98.6%。
**T103 Alert Chain smoke evidence 與 NoAlertsReceived2Hours 收斂（2026-05-20 台北）**：
- 觸發：T102 後 live 查證 `awoooi_alert_chain_last_success_timestamp` 已存在；真正問題是 smoke / rule 把主告警鏈 `alertmanager` 和外部事件來源 `sentry` / `signoz` 混在一起，導致 Sentry / SigNoz 兩小時沒有事件就被判成告警鏈斷線。另發現 `deploy-alerts.yaml` #1917 可 success 但 production rule 曾仍是舊 query，缺少內容落地驗證。
- 修正：`scripts/alert_chain_smoke_test.py` 改用 `urllib`，Alert Chain Metric 只驗 `awoooi_alert_chain_last_success_timestamp{source="alertmanager"}`，Prometheus 為第一證據源，必要時 fallback 到 API `/metrics` 並標示 evidence path。`NoAlertsReceived2Hours` 在 repo 與 k8s rule 全部限定 `source="alertmanager"`，Sentry / SigNoz 錯誤仍由 error-rate rules 覆蓋。
- 部署 gate：`scripts/ops/deploy-alerts.sh` 新增 repo/110 遠端 `alerts.yml`、`slo-rules.yml` SHA 比對；Prometheus reload 後讀 live `/api/v1/rules`，若 `NoAlertsReceived2Hours` query 沒有 `source="alertmanager"` 直接 fail，避免「workflow 綠燈但規則沒落地」再發生。
- 驗證：`py_compile` pass；`test_alert_chain_smoke_metric.py` 4 tests OK；alerts YAML parse ok；production smoke 8/8，Alert Chain Metric `evidence=prometheus` 且 alertmanager 0 分鐘前成功；Monitoring Coverage 100.0%、real_down=0；Prometheus live query 為 `time() - max by (source) (awoooi_alert_chain_last_success_timestamp{source="alertmanager"}) > 7200`，`ALERTS{alertname="NoAlertsReceived2Hours"}` 為空；110 remote hash 與 repo source match=yes。
- Production / CI：`598f33ae fix(monitoring): clarify alert chain smoke evidence`、`4956fbb8 fix(monitoring): verify alert rule deploy content` 已推 Gitea main；deploy marker `1b525b7c chore(cd): deploy 598f33a [skip ci]`。Actions：#1916 Code Review success 11s、#1917 Deploy Alert Rules success 22s、#1915 CD success 9m05s、#1918 Code Review success 11s、#1919 Deploy Alert Rules success 23s。
- 目前進度更新：AwoooP 告警可觀測鏈約 99.8%；Monitoring rule deploy proof 約 99.8%；低風險自動修復閉環約 95.2%；前端 AI 自動化管理介面同步約 99.6%；完整 AI 自動化管理產品化約 98.7%。
---
### 2026-04-20 晚 (台北) — C1-C4 全流程串接 — Playbook 鏈路保護（commit de2d34d）