43edff184d
C-1 Velero: 已確認運作中（daily-awoooi-prod schedule, 13d, MinIO Available）
C-2 Host rsync 備份:
scripts/ops/backup-from-110.sh — 188 每日凌晨 1:00 rsync 備份 110
- Harbor registry data（最高優先）
- Gitea repos
- bitan-pharmacy.git（若存在）
- 成功寫入 /var/run/backup-110.last_success 供 Prometheus 監控
- 失敗時 Telegram 告警
ops/monitoring/alerts-unified.yml — 新增 HostBackupFailed 告警規則
C-3 DR SOP 文件:
docs/runbooks/disaster-recovery/DR-K8s-awoooi.md (<15分鐘)
docs/runbooks/disaster-recovery/DR-Nginx.md (<5分鐘)
docs/runbooks/disaster-recovery/DR-Harbor.md (<30分鐘)
docs/runbooks/disaster-recovery/DR-Bitan.md (<5分鐘)
docs/runbooks/disaster-recovery/DR-Stock.md (<5分鐘)
部署備份腳本說明 (需手動執行):
scp scripts/ops/backup-from-110.sh ollama@192.168.0.188:~/bin/backup-from-110.sh
ssh ollama@192.168.0.188 "chmod +x ~/bin/backup-from-110.sh && mkdir -p /backup/110/{harbor,gitea}"
ssh ollama@192.168.0.188 "echo '0 1 * * * /home/ollama/bin/backup-from-110.sh' | crontab -"
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
48 lines
1.1 KiB
Markdown
48 lines
1.1 KiB
Markdown
# DR-Stock — stock 服務崩潰復原 SOP
|
||
|
||
**目標時間**: < 5 分鐘
|
||
**觸發場景**: stock K8s Service/Pod 無法訪問，或 NodePort 31235 無回應
|
||
**工具**: kubectl, ArgoCD
|
||
**最後更新**: 2026-04-11 (Claude Sonnet 4.6 Asia/Taipei)
|
||
|
||
---
|
||
|
||
## 快速復原
|
||
|
||
```bash
|
||
# 確認 stock Pod 狀態
|
||
kubectl get pods -n awoooi-prod -l app=stock 2>/dev/null || \
|
||
echo "stock 可能不在 K8s，確認是否為 Docker 服務"
|
||
|
||
# 若是 K8s Pod
|
||
kubectl rollout restart deployment/stock -n awoooi-prod
|
||
kubectl rollout status deployment/stock -n awoooi-prod --timeout=60s
|
||
|
||
# 驗收
|
||
curl -s -o /dev/null -w '%{http_code}' https://stock.wooo.work
|
||
# 期望: 200
|
||
```
|
||
|
||
---
|
||
|
||
## NodePort 31235 確認
|
||
|
||
```bash
|
||
# 確認 NodePort Service 存在
|
||
kubectl get svc -n awoooi-prod | grep 31235
|
||
|
||
# 直連 NodePort 測試
|
||
curl -s -o /dev/null -w '%{http_code}' http://192.168.0.121:31235
|
||
```
|
||
|
||
---
|
||
|
||
## 資料完整性確認
|
||
|
||
stock 資料存放在 PostgreSQL。確認資料庫連接正常：
|
||
|
||
```bash
|
||
# 透過 API 健康檢查確認 DB 正常
|
||
curl http://192.168.0.121:32334/api/v1/health | python3 -m json.tool
|
||
```
|