docs(monitoring): record t103 alert chain evidence rollout [skip ci]
This commit is contained in:
Your Name
@@ -1,3 +1,72 @@
|
||||
## 2026-05-20｜T103 Alert Chain smoke evidence 與 NoAlertsReceived2Hours 收斂
|
||||
|
||||
**觸發**：
|
||||
|
||||
- T102 留下 `awoooi_alert_chain_last_success_timestamp` non-critical 技術債；live 查證後確認指標其實已存在，但舊 smoke / rule 對「主告警鏈」與「Sentry / SigNoz 外部事件來源」的語意混在一起。
|
||||
- Prometheus 當時對 `source=sentry` / `source=signoz` 觸發 `NoAlertsReceived2Hours`，但 Sentry / SigNoz 並不保證每 2 小時都有事件；安靜不等於告警鏈斷線。
|
||||
- `deploy-alerts.yaml` #1917 顯示 success，但 110 `/home/wooo/monitoring/alerts.yml` 與 Prometheus live rule 一度仍是舊 query，代表告警規則部署缺少「內容已真的落地」的證據 gate。
|
||||
|
||||
**修正**：
|
||||
|
||||
- `scripts/alert_chain_smoke_test.py` 改用 Python 標準庫 `urllib.request`，移除 `requests` 隱性依賴。
|
||||
- Alert Chain Metric smoke 改查 `awoooi_alert_chain_last_success_timestamp{source="alertmanager"}`；Prometheus 是第一證據源，若 Prometheus 尚未 scrape 到，才 fallback 到 API `/metrics`，並明確標出 `evidence=prometheus` / `evidence=app_metrics`。
|
||||
- `NoAlertsReceived2Hours` 在 `ops/monitoring/alerts.yml`、`ops/monitoring/alerts-unified.yml`、`k8s/monitoring/alert-chain-monitor.yaml` 限定 `source="alertmanager"`；Sentry / SigNoz 的真正鏈路錯誤仍由 error-rate 規則處理，不再把「沒有事件」當成鏈路壞掉。
|
||||
- `scripts/ops/deploy-alerts.sh` 新增部署後驗證：比對 repo `alerts-unified.yml` / `slo-rules.yml` 與 110 遠端檔案 SHA，Prometheus reload 後再讀 live `/api/v1/rules`，若 `NoAlertsReceived2Hours` 沒有限定 `source="alertmanager"` 直接 fail。
|
||||
|
||||
**邊界**：
|
||||
|
||||
- 本輪沒有強迫 Sentry / SigNoz 產生 synthetic heartbeat，也沒有關掉 Gemini / Ollama / MCP 等 AI 路由。
|
||||
- 本輪沒有新增自動修復權限；它修的是告警判斷與部署驗證，避免 Operator 被假紅燈與假成功誤導。
|
||||
|
||||
**Local / live verification**：
|
||||
|
||||
```text
|
||||
python3 -m py_compile scripts/alert_chain_smoke_test.py apps/api/tests/test_alert_chain_smoke_metric.py
|
||||
-> pass
|
||||
python3 -m unittest apps.api.tests.test_alert_chain_smoke_metric -v
|
||||
-> 4 tests OK
|
||||
ruby -e YAML.load_file(...) for alerts.yml / alerts-unified.yml / alert-chain-monitor.yaml / deploy-alerts.yaml
|
||||
-> yaml ok
|
||||
python3 scripts/alert_chain_smoke_test.py --api-url https://awoooi.wooo.work --json
|
||||
-> PASSED, 8/8 checks; Alert Chain Metric: 最後 alertmanager 告警成功 0 分鐘前, evidence=prometheus
|
||||
python3 scripts/generate_monitoring.py --check --stabilization-sleep-seconds 0
|
||||
-> coverage 100.0%, real_down=0
|
||||
Prometheus /api/v1/rules NoAlertsReceived2Hours
|
||||
-> time() - max by (source) (awoooi_alert_chain_last_success_timestamp{source="alertmanager"}) > 7200
|
||||
Prometheus ALERTS{alertname="NoAlertsReceived2Hours"}
|
||||
-> []
|
||||
Prometheus direct silence query for source="alertmanager"
|
||||
-> []
|
||||
Remote hash check
|
||||
-> alerts.yml match=yes, slo-rules.yml match=yes
|
||||
```
|
||||
|
||||
**Gitea deploy / smoke**：
|
||||
|
||||
```text
|
||||
Code commits:
|
||||
598f33ae fix(monitoring): clarify alert chain smoke evidence
|
||||
4956fbb8 fix(monitoring): verify alert rule deploy content
|
||||
|
||||
Deploy marker:
|
||||
1b525b7c chore(cd): deploy 598f33a [skip ci]
|
||||
|
||||
Gitea Actions:
|
||||
1916 Code Review -> success in 11s
|
||||
1917 Deploy Alert Rules -> success in 22s
|
||||
1915 CD -> success in 9m05s
|
||||
1918 Code Review -> success in 11s
|
||||
1919 Deploy Alert Rules -> success in 23s
|
||||
```
|
||||
|
||||
**目前整體進度**：
|
||||
|
||||
- AwoooP 告警可觀測鏈：約 99.8%。
|
||||
- Monitoring rule deploy proof：約 99.8%。
|
||||
- 低風險自動修復閉環：約 95.2%。
|
||||
- 前端 AI 自動化管理介面同步：約 99.6%。
|
||||
- 完整 AI 自動化管理產品化：約 98.7%。
|
||||
|
||||
## 2026-05-20｜T102 Post-deploy monitoring coverage target freshness 穩定化
|
||||
|
||||
**觸發**：
|
||||
|
||||
@@ -2402,6 +2402,14 @@ Phase 6 完成後
|
||||
- Production / CI：`8fa8d690 fix(monitoring): stabilize post-deploy target coverage`、`6e5d68ee test(monitoring): avoid script bytecode cleanup noise`、`89f39759 ci: clean b5 test bytecode artifacts` 已推 Gitea main；deploy marker `f542aa52 chore(cd): deploy 6e5d68e [skip ci]`。Actions：#1911 Code Review success 10s、#1910 CD success 7m17s、#1913 Code Review success 10s、#1912 CD success 9m20s、#1914 Code Review success 12s。Post-CD smoke：health healthy/prod/mock_mode=false；Monitoring Coverage Check coverage 100.0%、real_down_jobs=0、`awoooi-api` UP；Playwright E2E 5 passed。
|
||||
- 目前進度更新：AwoooP 告警可觀測鏈約 99.7%；Monitoring coverage gate 準確性約 99.7%；低風險自動修復閉環約 95.2%；前端 AI 自動化管理介面同步約 99.6%；完整 AI 自動化管理產品化約 98.6%。
|
||||
|
||||
**T103 Alert Chain smoke evidence 與 NoAlertsReceived2Hours 收斂（2026-05-20 台北）**：
|
||||
- 觸發：T102 後 live 查證 `awoooi_alert_chain_last_success_timestamp` 已存在；真正問題是 smoke / rule 把主告警鏈 `alertmanager` 和外部事件來源 `sentry` / `signoz` 混在一起，導致 Sentry / SigNoz 兩小時沒有事件就被判成告警鏈斷線。另發現 `deploy-alerts.yaml` #1917 可 success 但 production rule 曾仍是舊 query，缺少內容落地驗證。
|
||||
- 修正：`scripts/alert_chain_smoke_test.py` 改用 `urllib`，Alert Chain Metric 只驗 `awoooi_alert_chain_last_success_timestamp{source="alertmanager"}`，Prometheus 為第一證據源，必要時 fallback 到 API `/metrics` 並標示 evidence path。`NoAlertsReceived2Hours` 在 repo 與 k8s rule 全部限定 `source="alertmanager"`，Sentry / SigNoz 錯誤仍由 error-rate rules 覆蓋。
|
||||
- 部署 gate：`scripts/ops/deploy-alerts.sh` 新增 repo/110 遠端 `alerts.yml`、`slo-rules.yml` SHA 比對；Prometheus reload 後讀 live `/api/v1/rules`，若 `NoAlertsReceived2Hours` query 沒有 `source="alertmanager"` 直接 fail，避免「workflow 綠燈但規則沒落地」再發生。
|
||||
- 驗證：`py_compile` pass；`test_alert_chain_smoke_metric.py` 4 tests OK；alerts YAML parse ok；production smoke 8/8，Alert Chain Metric `evidence=prometheus` 且 alertmanager 0 分鐘前成功；Monitoring Coverage 100.0%、real_down=0；Prometheus live query 為 `time() - max by (source) (awoooi_alert_chain_last_success_timestamp{source="alertmanager"}) > 7200`，`ALERTS{alertname="NoAlertsReceived2Hours"}` 為空；110 remote hash 與 repo source match=yes。
|
||||
- Production / CI：`598f33ae fix(monitoring): clarify alert chain smoke evidence`、`4956fbb8 fix(monitoring): verify alert rule deploy content` 已推 Gitea main；deploy marker `1b525b7c chore(cd): deploy 598f33a [skip ci]`。Actions：#1916 Code Review success 11s、#1917 Deploy Alert Rules success 22s、#1915 CD success 9m05s、#1918 Code Review success 11s、#1919 Deploy Alert Rules success 23s。
|
||||
- 目前進度更新：AwoooP 告警可觀測鏈約 99.8%；Monitoring rule deploy proof 約 99.8%；低風險自動修復閉環約 95.2%；前端 AI 自動化管理介面同步約 99.6%；完整 AI 自動化管理產品化約 98.7%。
|
||||
|
||||
---
|
||||
|
||||
### 2026-04-20 晚 (台北) — C1-C4 全流程串接 — Playbook 鏈路保護（commit de2d34d）
|
||||
|
||||
Reference in New Issue
Block a user