feat(dr): Sprint C — Host rsync 備份 + DR SOP 文件
C-1 Velero: 已確認運作中(daily-awoooi-prod schedule, 13d, MinIO Available)
C-2 Host rsync 備份:
scripts/ops/backup-from-110.sh — 188 每日凌晨 1:00 rsync 備份 110
- Harbor registry data(最高優先)
- Gitea repos
- bitan-pharmacy.git(若存在)
- 成功寫入 /var/run/backup-110.last_success 供 Prometheus 監控
- 失敗時 Telegram 告警
ops/monitoring/alerts-unified.yml — 新增 HostBackupFailed 告警規則
C-3 DR SOP 文件:
docs/runbooks/disaster-recovery/DR-K8s-awoooi.md (<15分鐘)
docs/runbooks/disaster-recovery/DR-Nginx.md (<5分鐘)
docs/runbooks/disaster-recovery/DR-Harbor.md (<30分鐘)
docs/runbooks/disaster-recovery/DR-Bitan.md (<5分鐘)
docs/runbooks/disaster-recovery/DR-Stock.md (<5分鐘)
部署備份腳本說明 (需手動執行):
scp scripts/ops/backup-from-110.sh ollama@192.168.0.188:~/bin/backup-from-110.sh
ssh ollama@192.168.0.188 "chmod +x ~/bin/backup-from-110.sh && mkdir -p /backup/110/{harbor,gitea}"
ssh ollama@192.168.0.188 "echo '0 1 * * * /home/ollama/bin/backup-from-110.sh' | crontab -"
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -156,6 +156,24 @@ groups:
|
||||
summary: "Velero 超過 24 小時未成功備份"
|
||||
description: "最後一次成功備份超過 24 小時"
|
||||
|
||||
# Sprint C-2 Host rsync 備份告警 (2026-04-11 Claude Sonnet 4.6)
|
||||
# backup-from-110.sh 成功時寫入 /var/run/backup-110.last_success
|
||||
# node-exporter textfile collector 讀取此檔案暴露指標
|
||||
- alert: HostBackupFailed
|
||||
expr: time() - node_textfile_scrape_error{collector="backup_110"} == 0 or absent(node_textfile_scrape_error{collector="backup_110"}) or (time() - backup_110_last_success_timestamp > 90000)
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
layer: docker-188
|
||||
team: ops
|
||||
component: backup
|
||||
host: "188"
|
||||
auto_repair: "false"
|
||||
alert_category: host_resource
|
||||
annotations:
|
||||
summary: "188 Host 備份超過 25 小時未成功"
|
||||
description: "backup-from-110.sh 可能失敗,/backup/110 資料可能過舊"
|
||||
|
||||
# =========================================================================
|
||||
# 資料庫告警 (database_alerts)
|
||||
# =========================================================================
|
||||
|
||||
Reference in New Issue
Block a user