Some checks failed
CD Pipeline / workflow-shape (push) Successful in 1s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / build-and-deploy (push) Has been cancelled
CD Pipeline / tests (push) Has been cancelled
CD Pipeline / post-deploy-checks (push) Has been cancelled
95 lines
12 KiB
Markdown
95 lines
12 KiB
Markdown
# Telegram 監控告警路由總帳
|
||
|
||
> 狀態:`source_policy_ready_runtime_not_applied`
|
||
> 真相邊界:本表把「已證明現況」、「建議目的地」、「目前阻擋」、「source risk」與「receipt 強度」分開。建議不等於 live route;alias 不是 numeric chat ID;本變更沒有送 Telegram、讀 token/chat ID、改 runtime route 或部署。
|
||
|
||
## 判讀規則
|
||
|
||
- `visible label` 只描述使用者畫面中的名稱;`proven runtime alias` 必須另有 source/runtime receipt,兩者不得互相替代。
|
||
- `proven route` 只描述目前可由 source 或 durable receipt 證明的行為,不把畫面名稱或建議當成 live delivery。
|
||
- `recommended route` 是 canonical 產品責任分流;若 bot alias、chat alias 或 durable receipt 未證明,egress 仍是 `blocked`。
|
||
- `receipt strength`:`strong` 代表 durable DB receipt;`weak` 代表 console/source 行為;`none` 代表沒有可驗證的 delivery receipt。
|
||
- 未知 product/signal/severity/destination 一律 no-egress;不再沿用 ADR-093 的 unknown-to-shared-group fallback。
|
||
- `AwoooI SRE 戰情室` 只收 shared infrastructure、security、DR 與 P0/P1 incident lifecycle。raw P2/P3、marketing、成功 heartbeat、未驗證 recommendation 和 bot 對話均不得進入。
|
||
|
||
## 監控與告警來源總帳
|
||
|
||
| source truth | 數量 | 角色與目前判定 |
|
||
|---|---:|---|
|
||
| `ops/monitoring/alerts-unified.yml` | 23 groups / 145 alert declarations | production primary Prometheus rule source;產品訊號必須帶 namespaced canonical route labels |
|
||
| `ops/monitoring/alerts.yml` | 81 declarations | legacy duplicate source,不加入 production primary 總數;保留相同產品 labels 防止誤復用 |
|
||
| 11 個 specialized Prometheus / SignOz rule files | 112 declarations | K3s、DB、flywheel、MinIO/Kali、NVIDIA、Ollama、SLO、agent latency、SignOz 等專項訊號 |
|
||
| primary + specialized 合計 | 257 declarations / 226 unique alert names | 29 個 alert name 在兩個以上來源重複,extra declarations 31;重複不是多送授權,仍由 grouping/fingerprint/route policy 收斂 |
|
||
| AWOOOI API `alert_rules` | 29 matching/action rules | 用於分類與修復候選,不是新的 signal generator,不重複計入 257 |
|
||
| Telegram egress scanner | 805 files / 50 normalized gateway callsites | active direct Bot API 0;7 個 direct calls 全在 GitHub-frozen legacy source,禁止執行 |
|
||
|
||
目前可明確歸戶的產品告警如下;未列出的 shared infrastructure/security/DR/K3s/DB/host 規則屬 `awoooi`,但只有 P0/P1 allowlisted lifecycle 能進 SRE 群:
|
||
|
||
| product_id | 明確 signal / alert | canonical signal family | 現行 egress |
|
||
|---|---|---|---|
|
||
| `2026fifa` | repository 中沒有產品專屬 Prometheus rule | `data_freshness_and_recommendation` | blocked |
|
||
| `agent-bounty-protocol` | registry 有 A2A、traffic security、treasury/onboarding 3 lanes;沒有專屬 Prometheus rule | 對應 3 個 registry families | blocked,缺 durable receipt |
|
||
| `awooogo` | 無產品專屬 rule | `*` | disabled |
|
||
| `awoooi` | 145 primary + 112 specialized declarations 的 shared/core owner | shared infrastructure / security / DR / incident lifecycle | 僅 P0/P1 allowlisted route allowed |
|
||
| `bitan-pharmacy` | `BitanWoooWorkDown` | `container_and_product_health` | blocked |
|
||
| `clawbot-openclaw` | `OpenClawDown` | `raw_monitoring_alert` | blocked;不得借用 AWOOOI TYPE-3 |
|
||
| `momo-pro` | `MoWoooWorkDown`、`MomoScraperSuccessLow` | scheduler/sales、crawler/data freshness | blocked;configured destination 仍 `chat_not_found` |
|
||
| `stockplatform-v2` | `StockWoooWorkDown` | `market_data_and_recommendation_freshness` | blocked |
|
||
| `tsenyang-website` | `TsenyangWebsiteDown` | `public_site_and_agent_action_health` | blocked |
|
||
| `vibework` | 無專屬 rule;`VibeAIAgent` 不等於 standalone VibeWork | `*` | not implemented |
|
||
| `vtuber` | 無產品專屬 rule | `*` | disabled |
|
||
|
||
## Source-side route corrections
|
||
|
||
- Alertmanager 產品規則現在帶 `awoooi_product_id`、`awoooi_signal_family`、`awoooi_route_severity`;gateway 以這組明確 context 覆蓋 legacy `TYPE-3`,所以 blocked product 不會再落到 shared SRE。
|
||
- SignOz public receiver 尚無 source authentication,因此即使 payload 自稱 canonical labels 也固定 no-egress;Sentry(signature verified)與 HMAC generic webhook 缺完整 namespaced context 時一律 `UNKNOWN / blocked_no_egress`。Alertmanager 另要求 direct peer 與完整 forwarded chain 都是 private IP,不再信任單一可偽造的 forwarded value。
|
||
- `docker-compose`、production ConfigMap、docker-health scripts 與 7 個 Gitea workflows 的 11 個 raw destination defaults 已移除;runtime 只可由既有 secret/env alias 注入,缺值即 fail closed。白名單也不再內嵌 numeric default。
|
||
- 本次只改 source policy,沒有讀 token、呼叫 Bot API、改 Telegram 群組、送測試訊息或偽造 delivery receipt。
|
||
|
||
## 11 產品路由清冊
|
||
|
||
| 產品 | 目前 proven route | 建議 canonical route | blocked unknown / egress status | source risk | receipt strength |
|
||
|---|---|---|---|---|---|
|
||
| `2026fifa` | source-only monitoring surface;沒有 runtime Telegram binding | 建立專屬 product alias 後,只送 data freshness、recommendation build、public API failure | `blocked`:chat/bot alias 與 durable receipt 均未證明 | generic env 可能誤入其他產品 lane | `none` |
|
||
| `agent-bounty-protocol` | `VibeAIAgent` bot-to-group route 已證明,涵蓋 A2A/traffic/treasury/onboarding;durable receipt 仍缺 | `vibe_ai_agent_control_plane`,先收斂到 gateway 與 durable receipt 再開 egress | `blocked`:route 存在但 console/source log 不能代替 durable delivery receipt | 訊息標題仍可能把 Agent Bounty 與 standalone VibeWork 混為同一產品 | `weak`:console/source only |
|
||
| `awooogo` | 沒有 runtime Telegram binding,egress disabled | 維持 disabled;未來啟用需先建立獨立 product alias/receipt | `disabled` | 不得用 shared SRE 當缺設定 fallback | `none` |
|
||
| `awoooi` | `AwoooI SRE 戰情室` 與 TsenYang monitoring-owner receipt 已證明;`小O_小龍蝦`/`叫小賀` 是 visible label/source-config candidate,不當成 runtime alias proof | TsenYang bot 才能送 canonical shared infra/security/DR/P0-P1 lifecycle;OpenClaw/NemoTron 只可在原訊息 interaction context 下回覆 | shared allowlist `allowed`;raw P2/P3、business/FinOps、marketing `blocked` | 若未驗 sender bot ownership,AI bot 會繞過 canonical alert owner | `strong`(僅 shared monitoring owner) |
|
||
| `bitan-pharmacy` | source-only product/container signals;沒有 canonical runtime Telegram binding | 建立 Bitan-owned alias 後才允許 product health/failure | `blocked`:目的地、bot alias、receipt backend 未證明 | cross-product container signal 容易被誤當 AWOOOI shared infra | `none` |
|
||
| `clawbot-openclaw` | 目前使用 `tsenyangbot` private lane,形成跨產品 bot ownership drift | 建立 ClawBot-owned bot/chat binding;interactive reply 與 monitoring 分離 | `blocked`:現有 private route 不具 ClawBot canonical ownership | 共用 AWOOOI bot 會造成 polling ownership、回覆與告警歸戶混亂 | `none` |
|
||
| `momo-pro` | MOMO bot live;configured chat readback 回 `400 chat_not_found` | 修復 MOMO-owned destination 後,才承接 crawler/freshness、scheduler/sales pipeline | `blocked`:runtime destination unreachable,且無 durable receipt | `WOOO` 只是 private bot lane/inferred,不是已證明 MOMO product group | `none` |
|
||
| `stockplatform-v2` | 與 TSENYANG Website 共用 `wooowooowooobot` private lane | 建立 StockPlatform-owned destination,只送 market data freshness、pipeline/API failure | `blocked`:跨產品 shared-private ownership drift,無 durable receipt | 同一 private bot lane 無法清楚歸戶、抑噪與回寫 | `none` |
|
||
| `tsenyang-website` | 與 StockPlatform 共用 `wooowooowooobot` private lane | 建立 TSENYANG Website-owned destination;operator interaction 不承接 bulk raw alert | `blocked`:跨產品 shared-private ownership drift,無 durable receipt | site 與 market-data 告警混流,owner/verifier 不可辨 | `none` |
|
||
| `vibework` | standalone product 沒有 runtime Telegram binding;`VibeAIAgent` 屬 Agent Bounty | 保持 `not_implemented`;若實作需獨立 product_id、alias 與 receipt | `not_implemented` | 名稱相似會造成錯誤歸戶與重複告警 | `none` |
|
||
| `vtuber` | 沒有 runtime Telegram binding,egress disabled | 維持 disabled;未來啟用須先建立 VTuber-owned alias | `disabled` | shared monitor 辨識到 signal 不等於產品已有 sender | `none` |
|
||
|
||
## 群組與 BOT 的責任邊界
|
||
|
||
| visible label | proven runtime alias | canonical candidate | 可以承接 | 不可以承接 |
|
||
|---|---|---|---|---|
|
||
| AwoooI SRE 戰情室 | `awoooi_sre_war_room` | TsenYang monitoring owner | shared infra/security/DR、P0/P1 lifecycle | raw P2/P3、business/FinOps、marketing、成功 heartbeat、各產品 bulk warning |
|
||
| 小O_小龍蝦 | `none`(visible label only) | `openclaw_bot` source-config candidate | 只在原 inbound message/thread context 下做診斷回覆 | canonical production alert ownership、root alert fanout |
|
||
| 叫小賀 | `none`(source-config only) | `nemotron_bot` candidate | 只在原 inbound message/thread context 下做 challenger/shadow reply | canonical production alert ownership |
|
||
| TsenYang private | `none`(visible label only) | `wooowooowooobot_private` candidate | 無;僅保留 ownership drift evidence | 在 ownership reconciliation 前不得視為任何產品 canonical route |
|
||
| tsenyangbot private | `none`(source-config only) | `tsenyangbot_private` candidate | 無;目前 ClawBot traffic 只作 drift evidence | 跨產品 monitoring ownership |
|
||
| WOOO | `none`(visible label only) | `wooo_private_bot_lane` candidate | 尚未證明任何 product-group ownership | 不得當作 MOMO 或 eWoooc canonical product group |
|
||
| VibeAIAgent | `none`(source-config only) | `vibe_ai_agent_control_plane` candidate | Agent Bounty A2A/traffic/treasury/onboarding(補 durable receipt 後) | standalone VibeWork 或其他產品告警 |
|
||
| MOMO configured chat | `none`(configured unreachable) | `momo_configured_chat_unresolved` candidate | 無;目前 `chat_not_found` | 在 runtime readback、receipt 與 verifier 通過前不得送 |
|
||
|
||
## Source risk 與落地缺口
|
||
|
||
1. ADR-093 的 unknown/default-to-group 與全產品治理 v2 衝突;legacy `resolve_chat_ids` 現在必須先通過 canonical registry。unknown、`TYPE-1` info/success 與 `TYPE-6B` business 全部回空 route,不再進 shared SRE。
|
||
2. Agent Bounty 的 bot-to-`VibeAIAgent` route 已證明,但 direct Bot API 與 console log 仍不是 durable receipt;因此 route 存在、egress 仍 blocked 兩件事必須同時呈現。
|
||
3. MOMO bot live 不等於 destination healthy;目前 `chat_not_found` 是明確 blocked runtime truth。`WOOO` 也只能標為 private/inferred lane,不能升格成 product group。
|
||
4. StockPlatform/TSENYANG Website 共用 private bot,ClawBot 又使用 AWOOOI-owned bot,都是 ownership drift;需各自 owner、gateway、receipt、verifier 後才能開 route。
|
||
5. Bitan、2026FIFA 是 source-only;AwoooGo、VibeWork、VTuber 是 disabled/not-implemented/no binding。它們不得因缺專屬群而 fallback 到 shared SRE。
|
||
6. Gateway 現在會在 provider call 前驗 product/signal/severity、destination alias 與 sender bot alias;非監控的 inbound command/callback/thread reply 使用獨立 interaction classifier,不與監控 allowlist 混用。
|
||
|
||
## Machine-readable truth
|
||
|
||
- Registry:`docs/security/telegram-canonical-routing-registry.snapshot.json`
|
||
- Schema:`docs/schemas/telegram_canonical_routing_registry_v1.schema.json`
|
||
- Resolver:`apps/api/src/services/notification_matrix.py`
|
||
- Tests:`apps/api/tests/test_telegram_canonical_routing_registry.py`
|
||
|
||
Production apply 前仍需同一 route change run 留下 source diff、check-mode、bounded apply、durable delivery receipt、post-verifier、rollback/no-write terminal 與 learning/writeback;本 source-only 交付不代表 runtime route 已更動。
|