fix(backup): aggregate receipt gap alerts
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 57s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 57s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled
This commit is contained in:
@@ -53852,3 +53852,24 @@ production browser smoke:
|
||||
|
||||
**下一步**:
|
||||
- 整合 Gitea main 最新 commit 後 push;CD 完成後讀回 Prometheus / Alertmanager / production scorecard,確認新 receipt metrics / rules 進 runtime,再往下一個 P0 blocker(99 VMware autostart / full Gitea dump / DB/settings/issues/packages/LFS / service green)推進。
|
||||
|
||||
## 2026-07-03 — P0-5 / P0-6 backup receipt runtime apply 與噪音收斂
|
||||
|
||||
**完成內容**:
|
||||
- Gitea main `0acda9f72 fix(backup): add alert receipt contract` 已經由後續 main deploy 帶入 runtime;CD `#4527` 讀回 job succeeded、deploy marker `374d4e4 -> dc4f0ec`、production deploy readback matched、success notification。
|
||||
- 受控同步新版 `backup-health-textfile-exporter.py` 到 110 `/home/wooo/scripts/` 與 188 `/home/ollama/scripts/`,均先備份舊檔;立即刷新 `backup_health.prom`,110 讀回 `awoooi_backup_alert_receipt_stage_expected_info` 88 條、188 讀回 12 條。
|
||||
- `scripts/ops/deploy-alerts.sh` dry-run 後正式套用:遠端規則備份、hash 驗證、Prometheus reload 與 key-rule verifier 成功;Prometheus 已載入 165 條規則。
|
||||
- runtime readback 確認 `BackupAlertReceiptMetricMissing` / `BackupAlertReceiptMetricMissing188` inactive,`BackupAlertReceiptStageMissing` rule health ok。初版逐 stage alert 會產生 100 條 pending,已立刻改成每 `host / receipt_channel` 一條聚合 alert,明細保留在 `awoooi_backup_alert_receipt_stage_fresh == 0`。
|
||||
|
||||
**驗證結果**:
|
||||
- `python3.11 -m pytest scripts/ops/tests/test_backup_health_textfile_exporter.py scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py ops/runner/test_cd_controlled_runtime_profile.py -q -p no:cacheprovider`:`65 passed`。
|
||||
- `python3.11 scripts/ops/backup-alert-label-contract-check.py`:`BACKUP_ALERT_LABEL_CONTRACT_OK`。
|
||||
- py_compile、Prometheus / baseline YAML parse、`guard-gitea-runner-pressure.py --root .`、`git diff --check`:通過。
|
||||
|
||||
**仍維持**:
|
||||
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth;沒有讀 Telegram token、runner token 或 chat raw payload。
|
||||
- 沒有使用 GitHub / gh / GitHub API / GitHub Actions。
|
||||
- 沒有重啟主機,沒有 Docker / Nginx / K3s / DB / firewall restart,沒有 workflow_dispatch,沒有 DROP / TRUNCATE / restore / prune,沒有送 Telegram 測試訊息。
|
||||
|
||||
**下一步**:
|
||||
- commit / push 聚合告警修正;再套用新版 Prometheus rule,確認 `BackupAlertReceiptStageMissing` pending/firing 數量聚合成 110 / 188 host-level summary,而不是逐 stage 噪音。
|
||||
|
||||
@@ -2604,6 +2604,7 @@ Prometheus 必須載入:
|
||||
判讀規則:
|
||||
|
||||
- `awoooi_backup_alert_receipt_stage_fresh == 0` 代表該 backup alert 的 `sent / received / dedup / escalation` 某段沒有新鮮證據;不能宣稱 Telegram/AwoooP receipt 完成。
|
||||
- `BackupAlertReceiptStageMissing` 必須聚合成每個 `host / receipt_channel` 一條 alert;明細用 `awoooi_backup_alert_receipt_stage_fresh{host="...",receipt_channel="..."} == 0` 查 `required_alert / scope / stage / required_notification_type`,避免每個 stage 都打 Telegram 造成噪音。
|
||||
- `required_alert` 是原始 backup alert 名稱;不得用 Prometheus 保留 label `alertname` 承載原始 alert,避免被 meta alert 覆蓋。
|
||||
- `required_notification_type` 是原始 backup alert 應走的類型;meta alert 自己的 `notification_type` 可維持 `TYPE-1`。
|
||||
- `scripts/ops/backup-alert-label-contract-check.py` 是 source rule contract;`scripts/ops/backup-alert-live-visibility-check.py` 是 Prometheus / Alertmanager live visibility contract。
|
||||
|
||||
@@ -413,7 +413,7 @@ scripts/ops/backup-alert-live-visibility-check.py --prometheus-url http://192.16
|
||||
判讀規則:
|
||||
|
||||
- `required_alert` 是原始 backup alert,例如 `BackupJobStale`、`BackupCoverageDomainStale`、`GiteaPrivateBundleBackupIncomplete`;不得用 Prometheus 保留 label `alertname` 承載它。
|
||||
- 每個 required alert 都要有 `sent / received / dedup / escalation` 四段 marker;缺一段就是 `BackupAlertReceiptStageMissing`。
|
||||
- 每個 required alert 都要有 `sent / received / dedup / escalation` 四段 marker;缺段明細由 `awoooi_backup_alert_receipt_stage_fresh == 0` 保留,`BackupAlertReceiptStageMissing` 只允許聚合成每個 `host / receipt_channel` 一條,避免 Telegram/AwoooP 噪音淹沒真問題。
|
||||
- marker 只能是 `/backup/alert-receipts/*.last_success` 這類 timestamp / evidence id;不得寫 Telegram token、raw payload、chat id 原文或 secret。
|
||||
- 這個 receipt proof 不取代真正 backup / restore proof;Gitea bundle、Gitea full dump、DB/settings/issues/packages/LFS、offsite、restore drill 仍要各自綠。
|
||||
|
||||
|
||||
Reference in New Issue
Block a user