fix(alertmanager): send direct alerts to sre group
All checks were successful
Code Review / ai-code-review (push) Successful in 10s

This commit is contained in:
OG T
2026-05-06 13:38:33 +08:00
parent eb71bc61ed
commit 4753099155
3 changed files with 50 additions and 8 deletions

View File

@@ -3581,3 +3581,45 @@ DockerContainerCpuSustainedHigh -> PB-20260505-F4197B
- 部署後觀察 24h`approval_records.matched_playbook_id IS NOT NULL` 必須從 0 開始增加。
- 若 guard 擋下大量 LLM 動作,下一步不是放寬 guard而是讓 PreDecision/MCP 先收 evidence再產生 domain-correct SSH/K8s action。
---
## 2026-05-06台北— Alertmanager 旁路改送 SRE 群組 + Sentry Snuba 修復
**觸發**Telegram 收到 `🚨 [Alertmanager Fallback] DockerContainerRestartSpike`,且訊息發到 OpenClaw 私訊/機器人對話;同一時間 AWOOOI 心跳正常,表示 fallback 旁路不是「API 離線才觸發」,而是 Alertmanager critical route 的 direct Telegram 旁路。
### 已修正
| 範圍 | 結果 |
|------|------|
| Alertmanager live config | `/home/wooo/monitoring/alertmanager.yml``telegram-direct.chat_id` 已從 `OPENCLAW_TG_CHAT_ID` 切到 `SRE_GROUP_CHAT_ID`,並以 HUP reload |
| Alertmanager repo template | `ops/alertmanager/alertmanager.yml` 改用 `SRE_GROUP_CHAT_ID_PLACEHOLDER`,避免後續部署回退到私訊 |
| Sentry / Snuba schema | 110 上 `/opt/sentry` 執行 `docker compose run --rm snuba-api bootstrap --force`,補齊 ClickHouse Snuba tables |
| Kafka offset | `ingest-consumer``ingest-events:0``generic-metrics-consumer``ingest-performance-metrics:0` reset 到 latest修正 `OffsetOutOfRange` |
### 驗證
```text
docker exec alertmanager amtool check-config /etc/alertmanager/alertmanager.yml
# SUCCESS
Alertmanager telegram-direct chat_id
# group/supergroup, suffix=74679
ClickHouse tables
# system.tables default count = 83
# default.errors_local = 1
# default.transactions_local = 1
# default.metrics_raw_v2_local = 1
Sentry consumers after reset
# events-consumer healthy
# generic-metrics-consumer healthy
# snuba-errors/metrics/transactions consumers healthy
# recent 45s logs: no OffsetOutOfRange / UNKNOWN_TABLE / ERROR markers
```
### 注意
- `DockerContainerRestartSpike` 使用 15 分鐘窗口,已發生的 restart spike 會在 Prometheus 窗口過去後退火;修復完成後短時間內 `ALERTS{alertname="DockerContainerRestartSpike"}` 仍可能暫時為 firing。
- 目前 Alertmanager 的 `telegram-direct` 仍是 critical 旁路不是真正「webhook failure conditional fallback」。後續應改成只由 AWOOOI API health / AlertChainDown 類告警觸發,避免 critical 告警在主鏈正常時雙路徑通知。

View File

@@ -293,7 +293,7 @@ For AI routing releases, also verify:
## 11. Immediate Next Items
1. Finish live deployment verification for `682c0b99` AwoooP route fix.
2. Create Wave 1 implementation checklist from this document.
3. Start with MCP Gateway bypass and platform approval authentication, because both directly affect safety and auditability.
4. Keep GCP-A/GCP-B/111 Ollama routing verification in every alert-path release until EffectivePolicy becomes authoritative.
1. Convert Alertmanager `telegram-direct` from "all critical bypass" to a true alert-chain/API-health-only emergency route; the 2026-05-06 hotfix moved it to `SRE_GROUP_CHAT_ID`, but it still duplicates critical alerts by design.
2. Continue Wave 1 with MCP Gateway bypass and MCP audit completeness, because production callers can still route around the gateway.
3. Keep GCP-A/GCP-B/111 Ollama routing verification in every alert-path release until EffectivePolicy becomes authoritative.
4. Add a Sentry/Snuba post-reboot health gate: ClickHouse table existence, Snuba migration status, and Kafka consumer offsets must be part of cold-start validation.