fix(backup): aggregate receipt gap alerts
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 57s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled

This commit is contained in:
Your Name
2026-07-03 07:55:55 +08:00
parent 468eba526e
commit 63003f7330
7 changed files with 44 additions and 41 deletions

View File

@@ -53852,3 +53852,24 @@ production browser smoke:
**下一步**
- 整合 Gitea main 最新 commit 後 pushCD 完成後讀回 Prometheus / Alertmanager / production scorecard確認新 receipt metrics / rules 進 runtime再往下一個 P0 blocker99 VMware autostart / full Gitea dump / DB/settings/issues/packages/LFS / service green推進。
## 2026-07-03 — P0-5 / P0-6 backup receipt runtime apply 與噪音收斂
**完成內容**
- Gitea main `0acda9f72 fix(backup): add alert receipt contract` 已經由後續 main deploy 帶入 runtimeCD `#4527` 讀回 job succeeded、deploy marker `374d4e4 -> dc4f0ec`、production deploy readback matched、success notification。
- 受控同步新版 `backup-health-textfile-exporter.py` 到 110 `/home/wooo/scripts/` 與 188 `/home/ollama/scripts/`,均先備份舊檔;立即刷新 `backup_health.prom`110 讀回 `awoooi_backup_alert_receipt_stage_expected_info` 88 條、188 讀回 12 條。
- `scripts/ops/deploy-alerts.sh` dry-run 後正式套用遠端規則備份、hash 驗證、Prometheus reload 與 key-rule verifier 成功Prometheus 已載入 165 條規則。
- runtime readback 確認 `BackupAlertReceiptMetricMissing` / `BackupAlertReceiptMetricMissing188` inactive`BackupAlertReceiptStageMissing` rule health ok。初版逐 stage alert 會產生 100 條 pending已立刻改成每 `host / receipt_channel` 一條聚合 alert明細保留在 `awoooi_backup_alert_receipt_stage_fresh == 0`
**驗證結果**
- `python3.11 -m pytest scripts/ops/tests/test_backup_health_textfile_exporter.py scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py ops/runner/test_cd_controlled_runtime_profile.py -q -p no:cacheprovider``65 passed`
- `python3.11 scripts/ops/backup-alert-label-contract-check.py``BACKUP_ALERT_LABEL_CONTRACT_OK`
- py_compile、Prometheus / baseline YAML parse、`guard-gitea-runner-pressure.py --root .``git diff --check`:通過。
**仍維持**
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth沒有讀 Telegram token、runner token 或 chat raw payload。
- 沒有使用 GitHub / gh / GitHub API / GitHub Actions。
- 沒有重啟主機,沒有 Docker / Nginx / K3s / DB / firewall restart沒有 workflow_dispatch沒有 DROP / TRUNCATE / restore / prune沒有送 Telegram 測試訊息。
**下一步**
- commit / push 聚合告警修正;再套用新版 Prometheus rule確認 `BackupAlertReceiptStageMissing` pending/firing 數量聚合成 110 / 188 host-level summary而不是逐 stage 噪音。

View File

@@ -2604,6 +2604,7 @@ Prometheus 必須載入:
判讀規則:
- `awoooi_backup_alert_receipt_stage_fresh == 0` 代表該 backup alert 的 `sent / received / dedup / escalation` 某段沒有新鮮證據;不能宣稱 Telegram/AwoooP receipt 完成。
- `BackupAlertReceiptStageMissing` 必須聚合成每個 `host / receipt_channel` 一條 alert明細用 `awoooi_backup_alert_receipt_stage_fresh{host="...",receipt_channel="..."} == 0``required_alert / scope / stage / required_notification_type`,避免每個 stage 都打 Telegram 造成噪音。
- `required_alert` 是原始 backup alert 名稱;不得用 Prometheus 保留 label `alertname` 承載原始 alert避免被 meta alert 覆蓋。
- `required_notification_type` 是原始 backup alert 應走的類型meta alert 自己的 `notification_type` 可維持 `TYPE-1`
- `scripts/ops/backup-alert-label-contract-check.py` 是 source rule contract`scripts/ops/backup-alert-live-visibility-check.py` 是 Prometheus / Alertmanager live visibility contract。

View File

@@ -413,7 +413,7 @@ scripts/ops/backup-alert-live-visibility-check.py --prometheus-url http://192.16
判讀規則:
- `required_alert` 是原始 backup alert例如 `BackupJobStale``BackupCoverageDomainStale``GiteaPrivateBundleBackupIncomplete`;不得用 Prometheus 保留 label `alertname` 承載它。
- 每個 required alert 都要有 `sent / received / dedup / escalation` 四段 marker一段就是 `BackupAlertReceiptStageMissing`
- 每個 required alert 都要有 `sent / received / dedup / escalation` 四段 marker段明細由 `awoooi_backup_alert_receipt_stage_fresh == 0` 保留,`BackupAlertReceiptStageMissing` 只允許聚合成每個 `host / receipt_channel` 一條,避免 Telegram/AwoooP 噪音淹沒真問題
- marker 只能是 `/backup/alert-receipts/*.last_success` 這類 timestamp / evidence id不得寫 Telegram token、raw payload、chat id 原文或 secret。
- 這個 receipt proof 不取代真正 backup / restore proofGitea bundle、Gitea full dump、DB/settings/issues/packages/LFS、offsite、restore drill 仍要各自綠。