feat(dr): Sprint C — Host rsync 備份 + DR SOP 文件

C-1 Velero: 已確認運作中(daily-awoooi-prod schedule, 13d, MinIO Available)

C-2 Host rsync 備份:
  scripts/ops/backup-from-110.sh — 188 每日凌晨 1:00 rsync 備份 110
    - Harbor registry data(最高優先)
    - Gitea repos
    - bitan-pharmacy.git(若存在)
    - 成功寫入 /var/run/backup-110.last_success 供 Prometheus 監控
    - 失敗時 Telegram 告警
  ops/monitoring/alerts-unified.yml — 新增 HostBackupFailed 告警規則

C-3 DR SOP 文件:
  docs/runbooks/disaster-recovery/DR-K8s-awoooi.md  (<15分鐘)
  docs/runbooks/disaster-recovery/DR-Nginx.md        (<5分鐘)
  docs/runbooks/disaster-recovery/DR-Harbor.md       (<30分鐘)
  docs/runbooks/disaster-recovery/DR-Bitan.md        (<5分鐘)
  docs/runbooks/disaster-recovery/DR-Stock.md        (<5分鐘)

部署備份腳本說明 (需手動執行):
  scp scripts/ops/backup-from-110.sh ollama@192.168.0.188:~/bin/backup-from-110.sh
  ssh ollama@192.168.0.188 "chmod +x ~/bin/backup-from-110.sh && mkdir -p /backup/110/{harbor,gitea}"
  ssh ollama@192.168.0.188 "echo '0 1 * * * /home/ollama/bin/backup-from-110.sh' | crontab -"

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
OG T
2026-04-11 03:04:18 +08:00
parent a29e5e1de2
commit 43edff184d
7 changed files with 472 additions and 0 deletions

View File

@@ -156,6 +156,24 @@ groups:
summary: "Velero 超過 24 小時未成功備份"
description: "最後一次成功備份超過 24 小時"
# Sprint C-2 Host rsync 備份告警 (2026-04-11 Claude Sonnet 4.6)
# backup-from-110.sh 成功時寫入 /var/run/backup-110.last_success
# node-exporter textfile collector 讀取此檔案暴露指標
- alert: HostBackupFailed
expr: time() - node_textfile_scrape_error{collector="backup_110"} == 0 or absent(node_textfile_scrape_error{collector="backup_110"}) or (time() - backup_110_last_success_timestamp > 90000)
for: 10m
labels:
severity: warning
layer: docker-188
team: ops
component: backup
host: "188"
auto_repair: "false"
alert_category: host_resource
annotations:
summary: "188 Host 備份超過 25 小時未成功"
description: "backup-from-110.sh 可能失敗,/backup/110 資料可能過舊"
# =========================================================================
# 資料庫告警 (database_alerts)
# =========================================================================