feat(observability): add guarded SigNoz migration wave A
This commit is contained in:
@@ -54739,3 +54739,11 @@ production browser smoke:
|
||||
- Agent99 SRE routing self-test `17/17`;Telegram ingest self-test `16/16`,不寫 alert、不送 Telegram。
|
||||
- Agent99 tests `49 passed`;reboot-recovery suite `142 passed`;Windows staging contract/synthetic、JSON/YAML parse 通過。
|
||||
- 112 runtime scope 已關閉。全主機 program 尚未關閉:只剩 fresh all-host reboot window 的三項歷史 blocker,必須由下一次真實 600 秒 drill 驗證,不能以事後綠燈清除。
|
||||
## 2026-07-15 — P0-OBS-002 SigNoz organization / canonical route Wave A
|
||||
|
||||
- Current P0 固定為 `P0-OBS-002`;organization、canonical data route、Foundry/OpAMP、ClickHouse upgrade 拆成獨立 run,禁止一次切換。官方 self-hosted OTLP 4317/4318 不需 ingestion key,但首次 organization/root 需要 durable credential,因此本輪未建立帳號、未送 setup POST、未讀 secret/raw DB/raw volume。
|
||||
- 新增 `ops/signoz/organization-canonical-route-migration.yaml` 與 no-write `signoz-canonical-route-preflight.py`。Wave A source receipt 為 `source_ready=true`、`active_route=incumbent`、`writes_performed=0`;5 類 producer endpoint 全部保持 `.188`,只把 `.110` 加入 API allowlist 與 production NetworkPolicy 4317/4318 transition allow。
|
||||
- `verify-signoz-otel-ingestion.sh` 現可用 `OTLP_ENDPOINT` / `ROUTE_ID` 比較 route。`P0-OBS-002-20260715T003500+0800` 對 incumbent `.188`、`.120 -> .110 direct`、`.121 -> .110 direct` 三條 replay/shadow 都得到 HTTP 200、ClickHouse exact marker `1`、15 秒 queue zero/failure delta zero;這只證明 log data-plane challenger,不代表 security/org/traces gate 完成。
|
||||
- `.188` bridge supervisor source 新增 user-systemd unit、source-range、restart、filesystem/process hardening、legacy cron rollback 與 receipt-gated retirement。三次 bounded takeover 分別暴露 user-manager capability directive、ExecStartPre JSON quoting、listener readiness race;第三版 unit journal 已證明 `Started`,但 executor 於 socket ready 前判失敗。依三輪上限停止 apply,執行 `--supervisor-rollback`。
|
||||
- Rollback terminal:legacy process 與 reboot cron 已恢復;`.120/.121` 各自 OTLP HTTP 200,獨立 synthetic marker exact `1`,queue/failure delta `0`。Source 已把 takeover flag 提前並加入最多 10 秒 readiness loop,但本輪不再重試 runtime。
|
||||
- Focused source tests `17 passed`;bash syntax、Python compile、YAML preflight、`git diff --check` 通過。Runtime 仍為 `partial_degraded`:`setupCompleted=false`、direct ingress policy 未關閉、traces 不新鮮、bridge 尚未 supervised、filelog normalization degraded、Wave B 未授權。
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
| **2026-07-14 執行期修正** | `superseded_by=global_product_governance_v2`;`.188` SigNoz 主機配置已取代 |
|
||||
| **負責範圍 / 到期日** | `platform-observability` / 暫時 `.188:24318` 橋接於 `2026-08-15 Asia/Taipei` 到期 |
|
||||
| **替代方案 / 驗證器** | 正式後端 `.110:4318`;`scripts/reboot-recovery/verify-signoz-otel-ingestion.sh` |
|
||||
| **2026-07-15 P0-OBS-002** | organization 與 canonical route 拆成獨立 gate;Wave A 只做 dual allow / replay / supervisor source,不切 active route |
|
||||
|
||||
---
|
||||
|
||||
@@ -57,6 +58,12 @@ SigNoz OTEL Collector (Prometheus Receiver)
|
||||
|
||||
2026-07-14: `.188` 不再承載 SigNoz stateful runtime。相容 bridge 只允許 K3s
|
||||
`.120/.121` 來源,並必須在到期日前將所有產生端與 NetworkPolicy 改為直接 `.110`。
|
||||
|
||||
2026-07-15: `.110` self-hosted OTLP 不使用 ingestion key,且目前 4317/4318
|
||||
對內網的暴露面大於 `.188` bridge source range。因此 producer promotion 前新增獨立
|
||||
`direct_ingress_policy_gate`;organization 首位 Admin 需要 credential,屬 high/critical
|
||||
secret boundary,不得與資料平面切流綁成一次不可回滾變更。正式狀態機以
|
||||
`ops/signoz/organization-canonical-route-migration.yaml` 為準。
|
||||
```
|
||||
|
||||
---
|
||||
@@ -134,8 +141,13 @@ Warning/Error Event 全量保留。Normal/Scheduled/Pulling/Pulled/Created/Start
|
||||
|
||||
## 後續行動
|
||||
|
||||
1. SSH 到 .188 部署剩餘配置 (prometheus-config-phase-o.yaml / prometheus-remote-write-signoz.yaml / minio-kali-alerts.yaml / SigNoz rules)
|
||||
2. Wave D: NVIDIA Grafana Dashboard + 監控覆蓋率報告
|
||||
1. P0-OBS-002 Wave A:保留 `.188` active route,完成 machine-readable inventory、
|
||||
dual allow、route-specific replay 與 guarded bridge supervisor。
|
||||
2. organization/root bootstrap 使用 secret reference 且獨立授權;禁止 raw DB UPDATE、
|
||||
raw secret read 或 stateful volume restore。
|
||||
3. Wave B:direct ingress policy、logs/metrics/traces same-run shadow、單 producer canary、
|
||||
queue/normalization verifier 全綠後,才可逐步 promote 並退場 OTLP bridge。
|
||||
4. `.188:3301 -> .110:8080` UI bridge 是不同資產,不在 OTLP bridge retirement 範圍。
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user