539 lines
47 KiB
Markdown
539 lines
47 KiB
Markdown
# BACKUP-STATUS.md — 備份狀態總覽
|
||
|
||
> 2026-04-05 Claude Code: 首席架構師完整盤點 — 全服務全自動化 + 告警機制
|
||
> 備份中心：192.168.0.110 (`/backup/`) — Restic + latest-only retention + Google Drive/rclone offsite mirror
|
||
> 2026-06-12 Codex pre-reboot refresh: 110 cron / Google Drive rclone / Alertmanager / credential escrow / cold-start scorecard rechecked.
|
||
> 2026-06-12 Codex post-reboot refresh: 110 recovered, offsite latest-only verified, and remaining red gates narrowed to 120 config capture plus credential escrow evidence.
|
||
> 2026-06-12 Codex post-120 recovery refresh: 120 restored, backup aggregate / offsite / full cold-start green; DR still blocked only by credential escrow evidence.
|
||
> 2026-06-13 Codex live refresh: backup core remains green; DR still blocked only by credential escrow evidence.
|
||
> 2026-06-13 Codex post-CD refresh: backup/offsite/alert contracts remain green after deploy marker `e4a349bc`; global SSH trust guardrail held; DR still blocked only by credential escrow evidence.
|
||
> 2026-06-13 Codex escrow refresh: 13:10 live report confirms offsite/rclone/script readiness is green and only five non-secret credential escrow evidence markers remain missing.
|
||
> 2026-06-18 Codex cold-start refresh: full-stack service readiness is green after stale failed Job classification; backup core remains green; DR still blocked only by five credential escrow evidence markers.
|
||
> 2026-06-24 Codex Velero/exporter refresh: 188 MinIO / Velero backup freshness, 188 PostgreSQL / Redis exporters, 188 node-exporter, and 110 disk pressure are recovered; DR still blocked only by five credential escrow evidence markers and service full-green is blocked by MOMO data freshness.
|
||
> 2026-06-24 22:17 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO import-boundary fix is production-deployed, but full-stack remains blocked by MOMO data freshness.
|
||
> 2026-06-24 22:40 Codex MOMO source readback: scheduler / DB / import metadata confirm the full-stack blocker is missing upstream source data, not backup freshness; no manual import or Drive write was performed.
|
||
> 2026-06-24 23:04 Codex cold-start gate refresh: repo-side v1.42 dry-run now emits MOMO source-absence evidence and blocks with `188 momo source file absent while daily sales data stale`; backup/offsite remains green and live 110 script deployment is not claimed.
|
||
> 2026-06-24 23:15 Codex live-sync gate readback: read-only deploy parity check correctly blocks because repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; installer remains a live write requiring explicit approval.
|
||
> 2026-06-24 23:33 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; live cold-start still blocks only on MOMO source absence / data freshness, not backup.
|
||
> 2026-06-25 09:05 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; live cold-start is `PASS=87 WARN=1 BLOCKED=1` because MOMO business data is stale and MOMO Google Drive token metadata is missing.
|
||
> 2026-06-25 09:37 Codex MOMO deploy readback: MOMO `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`, Gitea Actions `cd.yaml #910` succeeded, and 188 host / `momo-scheduler` source now fail closed on Google Drive auth/API failure. Backup/offsite remains green; full-stack still blocks on MOMO data freshness and `escrow_missing=5`.
|
||
> 2026-06-25 10:23 Codex MOMO fail-closed live proof: 10:04 scheduler run recorded Google Drive auth failure as `❌ 自動匯入失敗` and sent Telegram failure notification successfully; cold-start remains `PASS=87 WARN=1 BLOCKED=1` because business data is stale beyond 3 days and Drive token metadata/writeback is not confirmed. Backup/offsite remains green and `escrow_missing=5` remains the DR blocker.
|
||
> 2026-06-25 10:35 Codex route / DB / backup refresh: direct public routes for AWOOOI API, IwoooS, VibeWork, AwoooGo, MOMO health, Stock, and Bitan are 200; backup remains 110 `13/13` and 188 `2/2` fresh; MOMO daily and monthly DB bounds still stop at `2026-06-17`; latest import job remains `56 completed`.
|
||
> 2026-06-25 19:17 Codex latest recovery readback: post-start quick check is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`; 110 backup `13/13 fresh failed=0`, 188 backup `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`; MOMO data freshness is recovered through `2026-06-24`; DR still blocked only by `escrow_missing=5`.
|
||
> 2026-06-25 19:35 Codex product-data gate refresh: backup/offsite remains green, but overall "all products/data latest" is blocked by StockPlatform `/api/v1/system/freshness` (`core_margin_short_daily_missing`, `ai_recommendations_stale`). This is not a backup failure; keep `escrow_missing=5` as the DR blocker and Stock freshness as a separate product-data blocker.
|
||
> 2026-06-25 20:11 Codex StockPlatform cron-source recovery: StockPlatform Gitea/live source is now `fb91aa4c6272469d1d26e0820169629eac17d28a`; six missing production cron entrypoints are restored; natural cron runs for source remediation, market index, price, margin, chips, and AI no longer fail from missing files. Backup/offsite remains green. Stock freshness still blocks because official 2026-06-25 margin-short data is pending and AI recommendations correctly stay on 2026-06-24; this is still not a backup or restore incident.
|
||
> 2026-06-25 20:25 Codex 110 CPU cleanup: two orphan StockPlatform headless Chrome process groups were cleared by targeted approved `SIGTERM`; no Docker/systemd/Nginx/K8s/DB/backup write occurred. Backup/offsite remains green, DR still blocked by `escrow_missing=5`, and Stock freshness remains the only hard product-data blocker.
|
||
> 2026-06-25 21:14 Codex full wrapper refresh: StockPlatform 21:00 `intelligence-sync` and 21:10 AI pipeline naturally caught up; `/api/v1/system/freshness` is `status=ok` with blockers `[]`. Backup/offsite remains 110 `13/13` and 188 `2/2` fresh, `core_blockers=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`; full-stack service/data result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, with only `escrow_missing=5` blocking DR complete.
|
||
> 2026-06-26 06:28 Codex隔日 backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `last_backup_all=2026-06-26 02:31:02`, `escrow_missing=5`; full-stack service/data result remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`.
|
||
> 2026-06-27 00:56 Codex backup core recovery: 188 `momo_pg_daily` was fresh but temporarily false-blocked by cron/config drift (`configured_missing_188=1`). 188 crontab was backed up to `/home/ollama/momo_backups/crontab-before-momo-pg-host-owned-20260627-001925.txt`, the daily MOMO PostgreSQL backup entry was restored to host-owned `/home/ollama/bin/momo-pg-backup.sh`, and the exporter now reports `awoooi_backup_job_configured{host="188",job="momo_pg_daily"} 1`. `backup-status` now reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `configured_missing_188=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; DR still blocked only by credential escrow evidence.
|
||
> 2026-06-27 02:42 Codex post-reboot revalidation: `post-reboot-readiness-summary.sh` remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED` with `SERVICE_GREEN=1`, `PRODUCT_DATA_GREEN=1`, `BACKUP_CORE_GREEN=1`, `HOST_188_HYGIENE_BLOCKED=0`, `STOCK_FRESHNESS_STATUS=ok`, and `ESCROW_MISSING_COUNT=5`. `dr-offsite-operator-checklist.sh --check` confirms `CORE_COLD_START_GREEN=1`, `RECOVERY_STATE=CORE_READY_DR_OFFSITE_PENDING`, live Prometheus `awoooi_recovery_core_ready=1`, and `awoooi_recovery_dr_offsite_ready=0`.
|
||
|
||
---
|
||
|
||
## 2026-06-27 00:56 Backup / Offsite / Escrow Live Status
|
||
|
||
Read-only and minimal-write evidence sources: 00:56 `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110, 188 crontab backup / controlled MOMO backup path correction, 188 textfile exporter refresh, post-start quick check at 00:57, and Prometheus recovery recording-rule readback.
|
||
|
||
- 110 backup health: `13/13 fresh failed=0`。
|
||
- 188 backup health: `2/2 fresh failed=0`。
|
||
- Integrity / configured blockers: `core_blockers=0`、`configured_missing_110=0`、`configured_missing_188=0`、`script_missing_110=0`、`script_missing_188=0`、`integrity_stale=0`。
|
||
- 188 MOMO backup config drift fix: crontab rollback file `/home/ollama/momo_backups/crontab-before-momo-pg-host-owned-20260627-001925.txt`; active cron now uses `/home/ollama/bin/momo-pg-backup.sh`; exporter reports `awoooi_backup_job_configured{host="188",job="momo_pg_daily"} 1`。
|
||
- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。
|
||
- Last aggregate backup: `2026-06-26 02:31:02`。
|
||
- Prometheus recovery rules: `awoooi_recovery_core_ready=1`、`awoooi_recovery_dr_offsite_ready=0`。
|
||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||
- Full-stack service state: `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。Post-start quick check `PASS=38 WARN=3 BLOCKED=0`；StockPlatform freshness `status=ok`；MOMO daily freshness `2|2026-06-24`。
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0. |
|
||
| 188 backup freshness | VERIFIED | 2/2 fresh, failed count 0. |
|
||
| 188 MOMO backup cron/config | VERIFIED | Active crontab uses `/home/ollama/bin/momo-pg-backup.sh`; `configured_missing_188=0`. |
|
||
| Offsite / GDrive freshness | VERIFIED | `offsite_fresh=1`, `rclone_gdrive_fresh=1`. |
|
||
| Backup core blockers | GREEN | `core_blockers=0`; Prometheus `awoooi_recovery_core_ready=1`. |
|
||
| Full-stack service state | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | `POST_START_QUICK_CHECK PASS=38 WARN=3 BLOCKED=0`; service/data/backup core green. |
|
||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||
|
||
## 2026-06-26 06:28 Backup / Offsite / Escrow Live Status
|
||
|
||
Read-only evidence sources: 06:26 / 06:28 `post-start-quick-check.sh`, delegated `/backup/scripts/backup-status.sh --no-notify --no-refresh`, route-only wrapper retry validation, and direct StockPlatform / MOMO freshness readback.
|
||
|
||
- 110 backup health: `13/13 fresh failed=0`。
|
||
- 188 backup health: `2/2 fresh failed=0`。
|
||
- Integrity / configured blockers: `core_blockers=0`、`configured_missing_110=0`、`configured_missing_188=0`、`script_missing_110=0`、`script_missing_188=0`、`integrity_stale=0`。
|
||
- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。
|
||
- Last aggregate backup: `2026-06-26 02:31:02`。
|
||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||
- Full-stack service state: `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。Cold-start `PASS=89 WARN=0 BLOCKED=0`；StockPlatform freshness `status=ok`；MOMO daily freshness `1|2026-06-24`。
|
||
- Route note: 06:26 full wrapper had one-time route `000` for IwoooS / VibeWork, but direct curl and route-only wrapper immediately returned `200` and `RESULT=GREEN`; v1.6 wrapper now retries routes before blocking.
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0. |
|
||
| 188 backup freshness | VERIFIED | 2/2 fresh, failed count 0. |
|
||
| Offsite / GDrive freshness | VERIFIED | `offsite_fresh=1`, `rclone_gdrive_fresh=1`. |
|
||
| Backup core blockers | GREEN | `core_blockers=0`. |
|
||
| Full-stack service state | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | Cold-start `PASS=89 WARN=0 BLOCKED=0`; core wrapper `PASS=15 WARN=2 BLOCKED=0`; route-only wrapper `PASS=31 WARN=0 BLOCKED=0`. |
|
||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||
|
||
## 2026-06-25 19:17 Backup / Offsite / Escrow Live Status
|
||
|
||
Read-only evidence sources: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 19:17 Asia/Taipei, plus 19:05 post-start quick check and 19:05-19:06 route stability readback.
|
||
|
||
- 110 backup health: `13/13 fresh failed=0`。
|
||
- 188 backup health: `2/2 fresh failed=0`。
|
||
- Integrity / configured blockers: `core_blockers=0`、`dr_warnings=5`、`configured_missing_110=0`、`configured_missing_188=0`、`script_missing_110=0`、`script_missing_188=0`、`integrity_stale=0`。
|
||
- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。
|
||
- Last aggregate backup: `2026-06-25 02:35:09`。
|
||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||
- Full-stack service state: `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。這代表服務面、路由、K3s、MOMO data freshness、backup/offsite 為 green；不是 DR complete。
|
||
- MOMO DB readback from 19:05 wrapper: `daily_sales_snapshot=109061|2025-07-01|2026-06-24`、`DB_MONTHLY_SYNC 15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`、`DB_DAILY_FRESHNESS 1|2026-06-24`、latest import job `57 completed|即時業績_當日.xlsx|2026-06-25T13:16:47.359958|2026-06-25T13:18:02.964985|15383|15383|0`。
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0. |
|
||
| 188 backup freshness | VERIFIED | 2/2 fresh, failed count 0. |
|
||
| Offsite / GDrive freshness | VERIFIED | `offsite_fresh=1`, `rclone_gdrive_fresh=1`. |
|
||
| Backup core blockers | GREEN | `core_blockers=0`. |
|
||
| MOMO data freshness | VERIFIED | `DB_DAILY_FRESHNESS 1|2026-06-24`; latest job 57 completed cleanly. |
|
||
| Full-stack service state | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 21:14 `POST_START_QUICK_CHECK PASS=38 WARN=2 BLOCKED=0`; cold-start `PASS=89 WARN=0 BLOCKED=0`; StockPlatform freshness is OK, and only `escrow_missing=5` blocks DR complete. |
|
||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||
|
||
## 2026-06-25 20:11 StockPlatform Cron Source / Backup Boundary
|
||
|
||
Read-only and minimal-write evidence sources: StockPlatform Gitea / live source readback, one fast-forward `git pull --ff-only origin main` on 110 `/home/wooo/stockplatform-v2`, natural cron logs, `ops.job_runs`, and `/api/v1/system/freshness`.
|
||
|
||
- Backup remains green from the 19:17 readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`。
|
||
- DR blocker remains `escrow_missing=5`。
|
||
- StockPlatform source-version drift is repaired: live `/home/wooo/stockplatform-v2` and Gitea `main` are `fb91aa4c6272469d1d26e0820169629eac17d28a`。
|
||
- Six previously missing production cron entrypoint scripts are present, and every `scripts/ops/*.sh` referenced by `install-production-cron.sh` exists on live source。
|
||
- Natural cron evidence after source sync:
|
||
- `source-remediation-queue` succeeded at 19:56 and 20:00.
|
||
- `market-index-ingestion` succeeded at 20:00.
|
||
- `price-ingestion` succeeded at 20:02.
|
||
- `margin-short-ingestion` succeeded at 20:05 but official 2026-06-25 margin-short data remained pending, with `row_count=0`.
|
||
- `chips-ingestion` succeeded at 20:06.
|
||
- `ai-recommendation-pipeline` succeeded at the cron/job layer at 20:10 and correctly blocked internally on `core_margin_short_daily_incomplete,official_margin_short_daily_official_pending`。
|
||
- Stock freshness remains separate from backup: `/api/v1/system/freshness` is still `blocked` with `core_margin_short_daily_missing` and `ai_recommendations_stale`。
|
||
- No backup restore, manual DB restore, Docker restart, Nginx reload, K8s action, firewall change, or secret read was performed to address StockPlatform.
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| Backup / offsite | VERIFIED | 19:17 backup readback remains green. |
|
||
| StockPlatform cron source | REPAIRED | Live and Gitea at `fb91aa4c6272469d1d26e0820169629eac17d28a`; missing entrypoints restored. |
|
||
| StockPlatform natural cron entrypoints | VERIFIED | 19:56-20:10 official schedule runs no longer fail with `script_exit_127`. |
|
||
| StockPlatform product data freshness | BLOCKED_EXTERNAL_SOURCE | Official 2026-06-25 margin-short source pending; AI recommendations stay stale by design. |
|
||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||
|
||
---
|
||
|
||
## 2026-06-25 10:35 Backup / Offsite / Escrow Live Status
|
||
|
||
Read-only evidence sources: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 10:35 Asia/Taipei; scheduler log proof at 10:04; cold-start rerun at 10:35; public route and DB readback at 10:35.
|
||
|
||
- 110 backup health: `13/13 fresh failed=0`。
|
||
- 188 backup health: `2/2 fresh failed=0`。
|
||
- Integrity / configured blockers: `core_blockers=0`、`dr_warnings=5`、`configured_missing_110=0`、`configured_missing_188=0`、`script_missing_110=0`、`script_missing_188=0`、`integrity_stale=0`。
|
||
- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。
|
||
- Last aggregate backup: `2026-06-25 02:35:09`。
|
||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||
- Full-stack service release blocker remains separate: cold-start `PASS=85 WARN=1 BLOCKED=1`，原因是 MOMO business data freshness stale (`MOMO_DAILY_FRESHNESS 8|2026-06-17`) plus Google Drive token metadata missing / writeback not confirmed。這不是 backup freshness failure。
|
||
- MOMO code boundary now covers both failure modes: `cd.yaml #904` makes monthly sync failure fail the import job and prevents Drive file movement; `cd.yaml #910` makes Drive auth/API failure return `success=false` instead of a no-file success.
|
||
- Live scheduler proof: 2026-06-25 10:04 `auto_import_task` logs `Google Drive 認證失敗: could not locate runnable browser`，then logs `❌ 自動匯入失敗` and sends Telegram failure notification successfully. Therefore the alert is now a correct failure signal, not a heartbeat / no-file false green.
|
||
- MOMO DB readback: `daily_sales_snapshot=104614|2025-07-01|2026-06-17`; current-month `realtime_sales_monthly=10936|2026/06/01|2026/06/17`; latest import job remains `56 completed` with `10936/10936/0` and no newer successful daily-sales import by 10:35.
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0. |
|
||
| 188 backup freshness | VERIFIED | 2/2 fresh, failed count 0. |
|
||
| Offsite / GDrive freshness | VERIFIED | `offsite_fresh=1`, `rclone_gdrive_fresh=1`. |
|
||
| Backup core blockers | GREEN | `core_blockers=0`. |
|
||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||
| MOMO Drive token metadata | WARN | Host and container token metadata paths are missing; no token content was read. |
|
||
| MOMO Drive auth false-green | FIX_DEPLOYED_AND_LIVE_PROVEN | Gitea Actions `cd.yaml #910` success; 188 host and scheduler container source include fail-closed marker; 10:04 scheduler cycle failed closed and sent failure notification. |
|
||
| Service full green | NO-GO | Blocked by MOMO source absence / data freshness; token metadata warning also requires owner-gated evidence. |
|
||
|
||
---
|
||
|
||
## 2026-06-24 23:33 Backup / Offsite / Escrow Live Status
|
||
|
||
Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 23:33 Asia/Taipei.
|
||
|
||
- 110 backup health: `13/13 fresh failed=0`。
|
||
- 188 backup health: `2/2 fresh failed=0`。
|
||
- Integrity / configured blockers: `core_blockers=0`、`dr_warnings=5`、`configured_missing_110=0`、`configured_missing_188=0`、`script_missing_110=0`、`script_missing_188=0`、`integrity_stale=0`。
|
||
- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。
|
||
- Last aggregate backup: `2026-06-24 02:28:39`。
|
||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||
- Full-stack service release blocker remains separate: cold-start `PASS=88 WARN=0 BLOCKED=1`，原因是 `188 momo source file absent while daily sales data stale` / `MOMO_DAILY_FRESHNESS 7|2026-06-17`；這不是 backup freshness failure。
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0. |
|
||
| 188 backup freshness | VERIFIED | 2/2 fresh, failed count 0. |
|
||
| Offsite / GDrive freshness | VERIFIED | `offsite_fresh=1`, `rclone_gdrive_fresh=1`. |
|
||
| Backup core blockers | GREEN | `core_blockers=0`. |
|
||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||
| Service full green | NO-GO | Blocked by MOMO source absence / data freshness, not by backup. |
|
||
|
||
---
|
||
|
||
## 2026-06-24 22:17 Backup / Offsite / Escrow Live Status
|
||
|
||
Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 22:17 Asia/Taipei.
|
||
|
||
- 110 backup health: `13/13 fresh failed=0`。
|
||
- 188 backup health: `2/2 fresh failed=0`。
|
||
- Integrity / configured blockers: `core_blockers=0`、`dr_warnings=5`、`configured_missing_110=0`、`configured_missing_188=0`、`script_missing_110=0`、`script_missing_188=0`。
|
||
- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。
|
||
- Last aggregate backup: `2026-06-24 02:28:39`。
|
||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||
- Full-stack service release blocker remains separate: cold-start `PASS=86 WARN=0 BLOCKED=1`，原因是 `MOMO_DAILY_FRESHNESS 7|2026-06-17`；這不是 backup freshness failure。
|
||
- MOMO code boundary is now production-deployed: Gitea Actions `cd.yaml #904` succeeded at commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`; 188 live source marker confirms monthly sync failure now fails the job and prevents Drive file movement.
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0. |
|
||
| 188 backup freshness | VERIFIED | 2/2 fresh, failed count 0. |
|
||
| Offsite / GDrive freshness | VERIFIED | `offsite_fresh=1`, `rclone_gdrive_fresh=1`. |
|
||
| Backup core blockers | GREEN | `core_blockers=0`. |
|
||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||
| MOMO code boundary | VERIFIED | `cd.yaml #904` success and 188 live import-service marker. |
|
||
| Service full green | NO-GO | Blocked by MOMO data freshness, not by backup. |
|
||
|
||
2026-06-24 22:40 MOMO source absence clarification:
|
||
|
||
- Scheduler `auto_import_task` recent runs report `file_count=0` and `imported_count=0` for the expected Drive folder / pattern.
|
||
- `import_config` remains `gdrive_folder_path=當日業績匯入` and `gdrive_file_pattern=即時業績_當日`。
|
||
- Latest valid import job `56` already completed with `sync_success=true` and bounds `2026-06-01..2026-06-17`。
|
||
- Repo-side cold-start v1.42 dry-run emits `MOMO_SOURCE_EMPTY_EVIDENCE_LINES 21`、`MOMO_IMPORT_CONFIG 當日業績匯入|即時業績_當日`、`MOMO_LATEST_IMPORT_JOB 56|completed|即時業績_當日.xlsx|2026-06-18T11:41:00.853176|2026-06-18T11:42:02.309425|10936|10936|0` and keeps the only hard blocker as source absence.
|
||
- 110 live monitor deployment is intentionally not claimed: `verify-cold-start-monitor-deploy.sh` reports repo hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` vs live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`.
|
||
- Therefore backup/offsite remains green while service full-green remains blocked by business data source absence. Do not run backup restore or DB restore to solve this symptom.
|
||
|
||
---
|
||
|
||
## 2026-06-24 21:33 Backup / Offsite / Escrow Live Status
|
||
|
||
Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 21:33 Asia/Taipei.
|
||
|
||
- 110 backup health: `13/13 fresh failed=0`。
|
||
- 188 backup health: `2/2 fresh failed=0`。
|
||
- Integrity / configured blockers: `core_blockers=0`、`configured_missing_110=0`、`configured_missing_188=0`、`script_missing_110=0`、`script_missing_188=0`。
|
||
- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。
|
||
- Last aggregate backup: `2026-06-24 02:28:39`。
|
||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||
- Full-stack service release blocker remains separate: cold-start `PASS=86 WARN=0 BLOCKED=1`，原因是 `MOMO_DAILY_FRESHNESS 7|2026-06-17`；這不是 backup freshness failure。MOMO 程式版本已是 `V10.653`，但業務資料仍未到今天。
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0. |
|
||
| 188 backup freshness | VERIFIED | 2/2 fresh, failed count 0. |
|
||
| Offsite / GDrive freshness | VERIFIED | `offsite_fresh=1`, `rclone_gdrive_fresh=1`. |
|
||
| Backup core blockers | GREEN | `core_blockers=0`. |
|
||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||
| Service full green | NO-GO | Blocked by MOMO data freshness, not by backup. |
|
||
|
||
---
|
||
|
||
## 2026-06-24 Velero / Exporter / Disk-Pressure Live Status
|
||
|
||
2026-06-24 06:35 refresh:
|
||
|
||
- 110 backup health remains fresh: 13 configured jobs, stale `0`, failed count `0`, config failed `0`。
|
||
- 188 backup health remains fresh: 2 configured jobs, stale `0`, missing cron/script `0`。
|
||
- 188 `node-exporter` textfile scrape is restored: Prometheus `up{job="node-exporter-188"}=1` and `awoooi_backup_health_monitor_up{host="188"}=1`。
|
||
- 188 PostgreSQL exporter and Redis exporter are restored: local metrics `pg_up=1` / `redis_up=1`; Prometheus sees `up{job="postgres-exporter"}=1`, `pg_up=1`, `up{job="redis-exporter"}=1`, `redis_up=1`。
|
||
- 188 MinIO endpoint is healthy on `192.168.0.188:9000`; 120 Velero `BackupStorageLocation/default` is `Available`。
|
||
- One-off Velero backup `reboot-recovery-202606240456` completed successfully; 110 backup-health textfile reports `awoooi_velero_latest_completed_backup_fresh=1`。
|
||
- `VeleroBackupNotRun`、`BackupHealthMonitorMissing188`、`PostgreSQLDown`、`RedisDown` and 110 disk-pressure alerts are resolved.
|
||
- 110 `/` disk use is reduced from `92%` to `73%` after Docker image/build-cache cleanup only. Docker volume prune remains forbidden without explicit owner approval.
|
||
- Credential escrow readback remains blocked: `ESCROW_MISSING_COUNT=5`。
|
||
- Full service green is still blocked by MOMO business data freshness: `MOMO_DAILY_FRESHNESS 6|2026-06-17`。
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0, config failed 0. |
|
||
| 188 backup freshness | VERIFIED | 2/2 fresh, node-exporter scrape and textfile metrics restored. |
|
||
| Velero / MinIO storage | VERIFIED | MinIO health OK, BSL `Available`, backup `reboot-recovery-202606240456` `Completed`, freshness metric `1`. |
|
||
| PostgreSQL / Redis exporters | VERIFIED | `pg_up=1`, `redis_up=1`, Prometheus scrape `up=1` for both exporters. |
|
||
| Alert chain | VERIFIED_WITH_EXPECTED_REDLIGHTS | Exporter/Velero/disk alerts resolved; escrow missing and MOMO freshness blocker remain visible. |
|
||
| Credential escrow | BLOCKED | Five non-secret evidence markers still missing. |
|
||
| DR closeout | NO-GO | Must not be declared complete until real owner-provided non-secret evidence IDs are validated and markers are written. |
|
||
|
||
Operational helpers:
|
||
|
||
```bash
|
||
# 188 PostgreSQL / Redis exporters
|
||
ssh ollama@192.168.0.188 'bash /home/ollama/bin/188-db-exporters-restore.sh'
|
||
|
||
# 188 MinIO / 120 Velero BSL readback
|
||
ssh wooo@192.168.0.110 '/home/wooo/scripts/188-minio-velero-restore.sh'
|
||
|
||
# Maintenance-window one-off Velero backup + backup-health textfile refresh
|
||
ssh wooo@192.168.0.110 'CREATE_VELERO_BACKUP=true REFRESH_BACKUP_HEALTH=true /home/wooo/scripts/188-minio-velero-restore.sh'
|
||
```
|
||
|
||
Current policy: restore backup and monitoring red lights first; do not silence `VeleroBackupNotRun` or exporter-down alerts. Healthy heartbeat success messages are suppressed separately and should not be confused with real backup/data/escrow alerts.
|
||
|
||
---
|
||
|
||
## 2026-06-18 Cold-Start / Backup Live Status
|
||
|
||
2026-06-18 13:43 refresh:
|
||
|
||
- `full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1`：`PASS=84 WARN=0 BLOCKED=0`，result `GREEN`。
|
||
- K8s schedule evidence：`FAILED_JOBS=1`、`STALE_FAILED_JOBS=1`、`ACTIVE_FAILED_JOBS=0`、`BAD_PODS=0`。Retained `km-vectorize` failed Job is historical evidence only; active failed Job count is zero.
|
||
- 110 backup health：`total=13 stale=0 missing_cron=0 missing_script=0 failed_count=0 config_failed=0 integrity_total=2 integrity_stale=0`。
|
||
- 188 backup health：`total=2 stale=0 missing_cron=0 missing_script=0`。
|
||
- Public routes / TLS, momo DB parity, 120 / 121 K3s readiness, AWOOOI API/Web route checks all passed in the same cold-start run.
|
||
- Credential escrow readback remains blocked: `/backup/scripts/offsite-escrow-evidence-report.sh --no-color` reports `SCRIPT_MISSING_COUNT=0`、`OFFSITE_CONFIGURED=1`、`RCLONE_CONFIGURED=1`、`READINESS_REQUIRE_CONFIGURED_BLOCKED=0`、`ESCROW_MISSING_COUNT=5`。
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| Full cold-start service readiness | GREEN | `PASS=84 WARN=0 BLOCKED=0`; stale failed Job evidence is separated from active failed Job blockers. |
|
||
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0, config failed 0, integrity stale 0. |
|
||
| 188 backup freshness | VERIFIED | 2/2 fresh, missing cron/script 0. |
|
||
| Credential escrow | BLOCKED | Five non-secret evidence markers still missing: `restic_repository_password`, `offsite_provider_credentials`, `break_glass_admin_credentials`, `dns_registrar_recovery`, `oauth_ai_provider_recovery`. |
|
||
| DR closeout | NO-GO | Must not be declared complete until real owner-provided non-secret evidence IDs are validated and markers are written. |
|
||
|
||
Current policy: service recovery and backup health can be green while DR is still blocked. Do not fake escrow markers, do not paste secrets into repo/chat, and do not silence escrow alerts.
|
||
|
||
---
|
||
|
||
## 2026-06-13 Post-CD Live Status
|
||
|
||
2026-06-13 01:26 / 01:28 refresh:
|
||
|
||
- `/backup/scripts/backup-status.sh --no-notify`：110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`integrity_stale=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`core_blockers=0`、`escrow_missing=5`、last aggregate `2026-06-12 15:54:40`。
|
||
- `/home/wooo/node_exporter_textfiles/offsite_full_sync_verify.prom`：`awoooi_backup_offsite_remote_verify_ok=1`、`awoooi_backup_offsite_full_verify_fresh=1`，13 個 repo 都是 `snapshot_count=1` 且 `snapshot_latest_only=1`。
|
||
- `backup-alert-live-visibility-check.py`：Prometheus 與 Alertmanager 皆可見 5 個 escrow gap active/firing alerts。
|
||
- Prometheus rules API：`BackupConfigCapturePartial`、`BackupAggregateRunFailed`、`BackupCredentialEscrowEvidenceMissing`、`ColdStartRecoveryBlocked`、`ColdStartHost120Unreachable` 全部存在且 health `ok`；目前只有 escrow gap rule 正確 firing，其餘 inactive。
|
||
- `backup-alert-label-contract-check.py`：本地 `ops/monitoring/alerts-unified.yml` 與 live Prometheus label contract 對齊，24 條 baseline backup alert rules 已載入。
|
||
- 這代表備份核心仍綠；剩餘紅燈仍是 DR credential escrow evidence，不是備份腳本或 offsite sync 失敗。
|
||
|
||
| Gate | Status | Evidence |
|
||
|------|--------|----------|
|
||
| 110 backup cron | VERIFIED | Live crontab still has `02:00 backup-all`, `03:00 sync-offsite-backups --mode sync`, `06:05 backup-status`, `07:20 verify-offsite-full-sync`; success is summarized once daily and not sent as noisy Telegram heartbeat. |
|
||
| Backup freshness | VERIFIED | 2026-06-13 01:26 status shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`; last aggregate backup completed `2026-06-12 15:54:40`. |
|
||
| 188 momo backup cron/exporter contract | VERIFIED | 188 crontab now runs `/home/ollama/bin/momo-pg-backup.sh`; exporter reports `awoooi_backup_job_configured{host="188",job="momo_pg_daily"} 1`, so `configured_missing_188=0`. |
|
||
| Google Drive/rclone remote latest-only | VERIFIED | 2026-06-13 01:28 textfile confirms all 13 remote repos have `snapshot_count=1` and `snapshot_latest_only=1`; latest scheduled verifier log at 2026-06-12 07:20 returned `REMOTE_LATEST_ONLY_OK=1`, `FULL_MARKER_FRESH=1`, `VERIFY_OK=1`, `FAILED=0`. |
|
||
| Offsite gate marker | VERIFIED | `/backup/offsite/enable-rclone-sync` present; full marker fresh and verifier wrote `/home/wooo/node_exporter_textfiles/offsite_full_sync_verify.prom`. |
|
||
| Backup alert rules | VERIFIED | 2026-06-13 01:27 live check confirms Prometheus and Alertmanager expose the active `BackupCredentialEscrowEvidenceMissing` gap alerts for the five missing items; Prometheus rules API has all five required alert names healthy; label contract check confirms all baseline backup alert rules are loaded. |
|
||
| Backup aggregate health | VERIFIED | 2026-06-12 15:54 `/backup/scripts/backup-all.sh` completed `13/13` successfully; Configs captured 120 / 121 / K8s workloads / K8s secrets / Velero from source `192.168.0.120`; 18:55 `core_blockers=0`. |
|
||
| Credential escrow | BLOCKED | Five evidence markers missing. Only write non-secret marker evidence with `/backup/scripts/mark-credential-escrow-verified.sh`. |
|
||
| Config backup capture | VERIFIED | 2026-06-12 15:54 Configs backup succeeded for `120-k3s-host-configs`, `121-k3s-host-configs`, `cluster-k8s-workloads`, `cluster-k8s-secrets`, and `cluster-velero-backups`; latest Configs snapshot `bee9ae22`. |
|
||
| Full cold-start | GREEN | 2026-06-13 01:26 read-only rerun: `PASS=83 WARN=0 BLOCKED=0`; result `GREEN`. |
|
||
| 110 -> 120 / 188 SSH trust | VERIFIED | Final trust repair backup `/home/wooo/.ssh/known_hosts.before-120-188-final-refresh.20260613-011949`; CD fix `80e6ec1a` uses `/home/wooo/.ssh/deploy_known_hosts`; post-deploy marker `e4a349bc` did not clobber global `known_hosts`, and 120 / 188 entries remain present. |
|
||
| 120 console handoff | CLOSED | 120 root filesystem was repaired from console/initramfs with offline fsck, booted at `2026-06-12 15:13`, SSH returned, root mounted `rw`, failed units `0`, and K3s `mon` returned `Ready`. |
|
||
| 2026-06-05 manual backup remediation | VERIFIED with aggregate blocker | 18:40 status: `stale110=none`, `stale188=none`, `configured_missing_188=0`; manual snapshots: AWOOOI `b7d5ee4e`, Gitea `ea641613`, Open-WebUI `d1147507`, ClawBot `73ead3cc`, AI artifacts `b1161ab8`. |
|
||
| 2026-06-06 credential escrow audit | BLOCKED | 15:03 report confirms scripts/config/rclone are present, but all five non-secret evidence markers are still missing; 15:06 safe dry-run checklist is documented below. |
|
||
|
||
Current policy: normal success should not create immediate Telegram noise. Failures and operator-action states must still alert; a single daily status summary runs at 06:05.
|
||
|
||
2026-06-13 post-CD closeout:
|
||
- 110 / 120 / 121 / 188 public/core service recovery is green.
|
||
- Backup aggregate, Google Drive/rclone offsite latest-only, and full cold-start are green after 120 recovery.
|
||
- Latest normal CD deploy preserved API/Web workload balancing and did not break cold-start SSH trust.
|
||
- Do not close DR scorecard until all five credential escrow evidence markers are written with non-secret evidence IDs.
|
||
|
||
---
|
||
|
||
## Credential Escrow Evidence Checklist
|
||
|
||
2026-06-13 13:10 live refresh:
|
||
|
||
- `/backup/scripts/mark-credential-escrow-verified.sh --status`：仍缺 `restic_repository_password`、`offsite_provider_credentials`、`break_glass_admin_credentials`、`dns_registrar_recovery`、`oauth_ai_provider_recovery`。
|
||
- `/backup/scripts/offsite-escrow-evidence-report.sh --no-color`：`SCRIPT_MISSING_COUNT=0`、`OFFSITE_CONFIGURED=1`、`RCLONE_CONFIGURED=1`、`READINESS_REQUIRE_CONFIGURED_BLOCKED=0`、`ESCROW_MISSING_COUNT=5`、`SUMMARY PASS=8 WARN=5 BLOCKED=0`。
|
||
- Owner request package: [CREDENTIAL-ESCROW-EVIDENCE-OWNER-REQUEST.md](../security/CREDENTIAL-ESCROW-EVIDENCE-OWNER-REQUEST.md)。
|
||
- 判定：備份核心與 offsite readiness 是 green；DR closeout 仍 blocked，直到五個 marker 以真實非敏感 evidence-id 寫入。
|
||
|
||
Credential escrow marker 只證明「復原資料已被人工驗證且可取回」，不能包含任何 secret。
|
||
|
||
| Item | Acceptable evidence-id | Forbidden |
|
||
|------|------------------------|-----------|
|
||
| `restic_repository_password` | Password manager item ID, sealed envelope ID, recovery checklist ID | Restic password, recovery code, secret URL |
|
||
| `offsite_provider_credentials` | Vault item ID for Google Drive/rclone or provider credential record | OAuth token, refresh token, application key |
|
||
| `break_glass_admin_credentials` | Break-glass credential record ID or sealed envelope ID | Admin password, SSH private key, OTP seed |
|
||
| `dns_registrar_recovery` | Registrar recovery checklist ID or vault item ID | Registrar password, recovery codes |
|
||
| `oauth_ai_provider_recovery` | Provider account recovery checklist ID or vault item ID | API key, token, client secret |
|
||
|
||
Safe flow after human verification:
|
||
|
||
```bash
|
||
# 1. Read current status; this does not expose secrets.
|
||
/backup/scripts/mark-credential-escrow-verified.sh --status
|
||
|
||
# 2. Validate the non-secret evidence-id first.
|
||
/backup/scripts/mark-credential-escrow-verified.sh \
|
||
--item <allowed-item> \
|
||
--evidence-id <non-secret-evidence-id> \
|
||
--dry-run
|
||
|
||
# 3. Only after dry-run passes, write the marker.
|
||
/backup/scripts/mark-credential-escrow-verified.sh \
|
||
--item <allowed-item> \
|
||
--evidence-id <non-secret-evidence-id>
|
||
|
||
# 4. Recheck DR metric.
|
||
/backup/scripts/offsite-escrow-evidence-report.sh --no-color
|
||
```
|
||
|
||
`<non-secret-evidence-id>` must be an existing external reference. Placeholder values such as `EVIDENCE_ID_FOR_*` or `VAULT-ITEM-ID` are rejected and must not be written.
|
||
|
||
---
|
||
|
||
## 備份全景圖（全部自動化）
|
||
|
||
| # | 資料類型 | 備份腳本 | 排程 | 最大損失 | 狀態 |
|
||
|---|---------|---------|------|---------|------|
|
||
| 1 | Gitea (DB + 倉庫) | `backup-gitea.sh` | 每日 02:00 | 24h | ✅ |
|
||
| 2 | MOMO PostgreSQL | `backup-momo.sh` | 每日 02:00 | 24h | ✅ |
|
||
| 3 | Harbor (Registry + DB) | `backup-harbor.sh` | 每日 02:00 | 24h | ✅ |
|
||
| 4 | **AWOOOI PostgreSQL (完整)** | **`backup-awoooi.sh`** | **每日 02:00** | **6h** | **✅** |
|
||
| 4h | **AWOOOI PostgreSQL (高頻)** | **`backup-awoooi-frequent.sh`** | **08/14/20:00** | **6h** | **✅** |
|
||
| 5 | **Langfuse (AI 追蹤/評測)** | **`backup-langfuse.sh`** | **每日 02:00** | **24h** | **✅** |
|
||
| 6 | **Monitoring (Prometheus/Grafana/Alertmanager)** | **`backup-monitoring.sh`** | **每日 02:00** | **24h** | **✅** |
|
||
| 7 | **SignOz (ClickHouse 追蹤/日誌)** | **`backup-signoz.sh`** | **每日 02:00** | **24h** | **✅** |
|
||
| 8 | **Open-WebUI (LLM 對話紀錄)** | **`backup-open-webui.sh`** | **每日 02:00** | **24h** | **✅** |
|
||
| 9 | **ClawBot Redis (狀態/快取)** | **`backup-clawbot.sh`** | **每日 02:00** | **24h** | **✅** |
|
||
| - | K8s 資源 (全命名空間) | Velero + MinIO | 每日 02:00 | 24h | ✅ |
|
||
|
||
**備份總控**：`/backup/scripts/backup-all.sh` v3.0 — 統一執行 9 個備份
|
||
|
||
---
|
||
|
||
## 告警機制
|
||
|
||
備份失敗與需要人工處理的狀態必須推送 AwoooP / Telegram。正常成功不即時推送，避免洗版；成功狀態由每日 06:05 摘要與 Prometheus/textfile 證據承載。
|
||
|
||
| 狀態 | Severity | Telegram 收到 |
|
||
|------|---------|--------------|
|
||
| `success` | info | 不即時洗版；每日 06:05 backup status 摘要 |
|
||
| `warning` | warning | ⚠️ 黃色警告 |
|
||
| `failed` | **critical** | 🔴 **立即告警** |
|
||
|
||
**告警端點**：`http://192.168.0.188:8088/api/v1/webhook/custom`
|
||
**測試指令**：
|
||
```bash
|
||
source /backup/scripts/common.sh
|
||
notify_clawbot "failed" "backup-test" "測試告警" 0
|
||
```
|
||
|
||
---
|
||
|
||
## 保留策略
|
||
|
||
2026-05-19 起，110 本地 restic repo、188 MOMO 檔案備份與 Google Drive/rclone 離機鏡像採 latest-only 策略：成功建立新 snapshot 後只保留最新一份。2026-06-13 01:28 live textfile 已確認 Google Drive/rclone remote 13 個 repo 各 1 份，且 latest-only 指標全為 `1`。
|
||
|
||
2026-06-04 manual refresh evidence:
|
||
- 188 `momo-pg-backup.sh` produced `momo_analytics_20260604_154234.sql.gz` and pruned old backups beyond keep-last=1.
|
||
- 110 `backup-awoooi-frequent.sh` completed restic snapshot `7440d75f` and pruned previous AWOOOI high-frequency DB snapshot.
|
||
- 18:54 `backup-status.sh --no-notify`: `stale110=none`, `stale188=none`, `configured_missing_188=0`, `core_blockers=1`, `escrow_missing=5`.
|
||
|
||
18:55 cold-start scorecard refresh:
|
||
- `PASS=71 WARN=3 BLOCKED=3`.
|
||
- Remaining hard blocks: 120 ping, 120 SSH, and 120 K3s read-only check.
|
||
- 188 backup health stale jobs are clear.
|
||
- momo current-month parity is green: `2215|2215|2026-06-01|2026-06-04|2026-06-01|2026-06-04`.
|
||
|
||
19:02 120 console handoff evidence:
|
||
- local/110/121/188 cannot reach 192.168.0.120.
|
||
- K3s node lease for `mon` stopped renewing at `2026-05-22 02:48:36 +08`.
|
||
- `120-fsck-maintenance-checklist.sh --no-color` returns `PASS=2 WARN=2 BLOCKED=3`, so backup aggregate remains correctly blocked until console/SSH recovery.
|
||
|
||
2026-06-12 18:55 update: 120 has returned and the aggregate backup blocker is cleared. `/backup/scripts/backup-all.sh` completed `13/13`, full offsite sync completed `13/13`, full verifier returned `REMOTE_LATEST_ONLY_OK=1` / `VERIFY_OK=1`, and `backup-status.sh --no-notify` reports `core_blockers=0`. The only remaining DR warning is `escrow_missing=5`.
|
||
|
||
2026-06-13 01:28 update: post-CD live readback still shows `remote_verify_ok=1`, `full_verify_fresh=1`, and all 13 repos `snapshot_count=1`; backup core remains green after deploy marker `e4a349bc`.
|
||
|
||
2026-06-05 manual remediation:
|
||
- 16:00 live check still had 120 unreachable, `stale110=awoooi_db`, `backup_all failed=6`, and `escrow_missing=5`.
|
||
- 14:00 AWOOOI high-frequency backup failed, then 16:01 manual rerun completed snapshot `b7d5ee4e`.
|
||
- 02:00 Gitea failure was caused by stale container `/tmp/gitea-dump.zip`; it was renamed in-container to `/tmp/gitea-dump.stale.20260605_161032.zip`, then Gitea backup completed snapshot `ea641613`.
|
||
- `scripts/backup/backup-gitea.sh` and live 110 `/backup/scripts/backup-gitea.sh` now preserve stale container dump files with timestamped names before starting a new dump.
|
||
- 110 -> 188 SSH known_hosts was refreshed after fingerprint match for `192.168.0.188`; Open-WebUI backup completed snapshot `d1147507`.
|
||
- ClawBot backup completed snapshot `73ead3cc`; BGSAVE still warned, but the Redis volume backup succeeded.
|
||
- AI artifacts backup completed snapshot `b1161ab8`.
|
||
- Full offsite sync was skipped by runway gate because the next scheduled backup was too close; partial sync for `awoooi gitea open-webui clawbot ai-artifacts` completed `5/5`.
|
||
- 18:39 full verifier confirmed all 13 Google Drive/rclone repos have `remote snapshots=1`, `REMOTE_LATEST_ONLY_OK=1`, and `VERIFY_OK=1`.
|
||
- 18:40 backup status still reports `failed=6` / `core_blockers=6` because the 02:00 aggregate history remains failed until `backup-all` reruns after 120 returns. Do not mark aggregate backup green from individual backup success alone.
|
||
|
||
2026-06-06 convergence evidence:
|
||
- 14:46 live check: 120 still ping/SSH failed and K3s `mon` remains `NotReady,SchedulingDisabled`.
|
||
- 02:00 aggregate backup failed only Configs: `全服務備份完成 (1532s) - 1 個失敗 (12/13)`.
|
||
- 14:58 `backup-status.sh --no-notify`: `stale110=none`, `stale188=none`, `failed=1`, `core_blockers=1`, `escrow_missing=5`.
|
||
- 14:46 `verify-offsite-full-sync.sh --write-textfile --no-color`: all 13 remote repos have one snapshot, `REMOTE_LATEST_ONLY_OK=1`, `VERIFY_OK=1`.
|
||
- 15:03 cold-start scorecard: `PASS=71 WARN=3 BLOCKED=3`; direct 188 checks still show `momo-scheduler` healthy with recent log activity, and the scheduler WARN is no longer present in the scorecard.
|
||
- 15:03 credential escrow report: rclone/offsite readiness is configured, but `restic_repository_password`, `offsite_provider_credentials`, `break_glass_admin_credentials`, `dns_registrar_recovery`, and `oauth_ai_provider_recovery` still lack non-secret evidence markers. Do not write placeholders or secrets.
|
||
|
||
---
|
||
|
||
## Crontab 完整排程（110）
|
||
|
||
```
|
||
0 2 * * * backup-all.sh ← 9 個服務完整備份
|
||
0 8,14,20 * * * backup-awoooi-frequent.sh ← AWOOOI 高頻（每 6 小時）
|
||
0 3 * * * sync-offsite-backups.sh --mode sync ← Google Drive/rclone gated sync
|
||
5 6 * * * backup-status.sh ← 每日一次備份狀態摘要，避免成功心跳洗版
|
||
20 7 * * * verify-offsite-full-sync.sh --write-textfile ← Google Drive/rclone latest-only 驗證
|
||
```
|
||
|
||
---
|
||
|
||
## 備份架構
|
||
|
||
```
|
||
192.168.0.110 (/backup/scripts/backup-all.sh) 每日 02:00
|
||
├── [1/9] backup-gitea.sh → gitea dump → /backup/gitea
|
||
├── [2/9] backup-momo.sh → SSH 188 pg_dump momo → /backup/momo
|
||
├── [3/9] backup-harbor.sh → harbor dump → /backup/harbor
|
||
├── [4/9] backup-awoooi.sh → SSH 188 pg_dump awoooi_prod/dev/k3s → /backup/awoooi
|
||
├── [5/9] backup-langfuse.sh → docker exec langfuse-db pg_dump → /backup/langfuse
|
||
├── [6/9] backup-monitoring.sh → volumes prometheus/grafana/alertmanager → /backup/monitoring
|
||
├── [7/9] backup-signoz.sh → volumes signoz-clickhouse/sqlite → /backup/signoz
|
||
├── [8/9] backup-open-webui.sh → SSH 188 volume open-webui → /backup/open-webui
|
||
└── [9/9] backup-clawbot.sh → SSH 188 volume clawbot-redis → /backup/clawbot
|
||
|
||
備份失敗 → notify_clawbot("failed") → /webhook/custom 或 AwoooP/Alertmanager path → Telegram 🔴
|
||
備份成功 → textfile / Prometheus / 06:05 status 摘要，不即時洗版
|
||
|
||
192.168.0.188 (Velero) 每日 02:00
|
||
└── K8s 資源快照 → MinIO :9000 (bucket: velero)
|
||
```
|
||
|
||
---
|
||
|
||
## 尚未備份（說明）
|
||
|
||
| 服務 | 原因 | 備記 |
|
||
|------|------|------|
|
||
| Prometheus TSDB | 原始指標數據（非設定），TSDB 自帶 30d TTL | 低優先；Grafana 設定已備份 |
|
||
| Sentry | 目前沒有在運行（docker ps 空）| 有 volume，重新部署後再評估 |
|
||
| Redis (AWOOOI) | Cache/WorkingMemory，無持久業務數據 | 低優先 |
|
||
| Velero MinIO 數據 | MinIO 是備份的備份，需離機備份 | 待評估 B2/S3 offsite |
|
||
|
||
---
|
||
|
||
## 驗證 SOP
|
||
|
||
```bash
|
||
# 最新備份日誌
|
||
ssh wooo@192.168.0.110 "tail -50 /backup/logs/backup.log"
|
||
|
||
# 所有服務快照數
|
||
ssh wooo@192.168.0.110 "for r in gitea momo harbor awoooi langfuse monitoring signoz open-webui clawbot; do
|
||
echo -n \"\$r: \"
|
||
restic -r /backup/\$r snapshots --password-file /backup/scripts/.restic-password 2>/dev/null | grep -c snapshot || echo 0
|
||
done"
|
||
|
||
# 告警測試
|
||
ssh wooo@192.168.0.110 "source /backup/scripts/common.sh && notify_clawbot 'warning' 'manual-test' '手動告警測試' 0"
|
||
```
|
||
|
||
---
|
||
|
||
## 相關文件
|
||
|
||
- [REBOOT-RECOVERY-SOP.md](REBOOT-RECOVERY-SOP.md) - 重開機恢復 SOP
|
||
- `scripts/backup/` - 所有備份腳本（Git 版本）
|
||
- `/backup/scripts/` (on 110) - 實際部署腳本
|