wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

fix(backup): restore reboot freshness readback
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Failing after 2m1s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details

Browse Source
This commit is contained in:
Your Name
2026-07-01 21:09:29 +08:00
parent 130480207a
commit d4c513a022
5 changed files with 261 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
docs/LOGBOOK.md
View File

@@ -1,3 +1,30 @@
## 2026-07-01 — 21:05 backup / DR freshness 收斂與高頻備份保護
**照主線修正的問題**：
- 188 `backup_from_110` stale：受控重跑 `/home/ollama/bin/backup-from-110.sh`，Harbor / Gitea / bitan backup 均成功；188 exporter 讀回 `backup_from_110 fresh=1`、Gitea private bundle `expected_repo_missing_count=0`、`failed_repo_count=0`。
- 110 DR phase 誤判：offsite full marker fresh 但 partial marker stale 時，`backup-health-textfile-exporter.py` 會錯誤退回 `run_small_dry_run_then_partial_sync`；已修成 full fresh + escrow missing 時 next step 固定 `complete_credential_escrow_review`，並新增回歸測試。
- 110 `awoooi_db` high-frequency backup stale：20:00 cron 因 `too many connections for role "awoooi"` 失敗。讀回 `awoooi` role 原本 `CONNECTION LIMIT 2`，當下 DB 舊 OR 查詢 `0`、Postgres CPU 已低，故走 controlled short window：臨時放寬到 `4`、解除 `statement_timeout` 跑一次備份，完成後恢復 role，再把正式保守值調成 `CONNECTION LIMIT 4`，保留 `statement_timeout=750ms`、`max_parallel_workers_per_gather=0`、`enable_seqscan=off`。
- 高頻備份腳本 source-of-truth 修復：`scripts/backup/backup-awoooi-frequent.sh` 移除舊硬編 DB password，改為只在腳本內部從 K8s Secret key preference 解析 `AWOOOI_BACKUP_DATABASE_URL` / `BACKUP_DATABASE_URL` / `DATABASE_URL`，使用臨時 `.pgpass`，並以 `PGOPTIONS` 讓 backup session 覆寫 role 上的 `statement_timeout`。live 110 已部署備份腳本，hash `01c5de7d08bb21d88aa39fd4d195e70131fa4335a8fc0b6e3f101d2bc9747142`。
**驗證**：
- `python3.11 -m pytest scripts/ops/tests/test_backup_health_textfile_exporter.py -q`：`3 passed`。
- `python3.11 -m py_compile scripts/ops/backup-health-textfile-exporter.py`：通過。
- `bash -n scripts/backup/backup-awoooi-frequent.sh`：通過。
- 110 high-frequency backup：`pg_dump 28G`，Restic snapshot `211a8948`，`2026-07-01T20:58:20+08:00`，tags `service:awoooi,freq:6h,timestamp:20260701_204344`；tmp dump 已清除。
- 110 backup exporter：`awoooi_backup_job_fresh{job="awoooi_db"} 1`、`awoooi_backup_dr_phase{next_step="complete_credential_escrow_review"} 3`、`awoooi_backup_dr_credential_escrow_missing_count 5`。
- `bash scripts/reboot-recovery/full-stack-recovery-scorecard.sh`：`CORE_COLD_START_WARN_GATES=3`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`NEXT_STEP=complete_credential_escrow_review`。
- `bash scripts/reboot-recovery/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1`：`PASS=93 WARN=3 BLOCKED=0`。
**仍維持 / 未完成**：
- 不可宣稱 full green 或 10 分鐘全自動恢復完成；剩餘 WARN 是 MOMO current-month/source freshness warning，以及 110 aggregate `backup-all` 上次仍有 failed components。
- `ESCROW_MISSING_COUNT=5` 仍是真正 DR evidence gate；不得偽造 marker 或把一般批准當成 credential escrow evidence。
- 高頻全量 `pg_dump` 這次耗時 `1152s` 並產生 `28G` dump，證明每 6 小時全量 DB dump 對 188 / 110 都太重。後續 P0 需建立專用 backup DB role / `AWOOOI_BACKUP_DATABASE_URL`，並把大型可重建 snapshot 表改為分層備份或 retention 後再納入 full backup；不能再只靠放寬 app role 當長期解法。
- 110 SSH control path 仍有 intermittent timeout，且 post-backup 讀回 load5 約 `10.41`、top 包含 `systemd` / `gitea` / ClickHouse / Kafka；服務可用但不得宣稱 110 control path 永久穩定。
**邊界**：未重啟主機，未 restart Docker / Nginx / K3s / DB / firewall，未讀 secret value / token / `.env` / raw sessions / SQLite / auth，未寫 credential escrow marker，未使用 GitHub / `gh` / GitHub API，未恢復 generic runner。
**下一步**：照 P0 繼續收斂剩餘 WARN：先處理 MOMO source freshness / current-month confirmation 與 110 `backup-all` failed components；另開受控設計把高頻全量 pg_dump 改成專用 backup role + 分層備份，避免下一次重啟或 cron 再製造 DB/IO 壓力。
## 2026-07-01 — 20:34 P0-006 reboot SLO machine-readback source closure
**照主線修正的問題**：

4
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.87
> Version: v1.88
> Last updated: 2026-07-01 Asia/Taipei
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
@@ -18,6 +18,8 @@ v1.79 active owner response template rule：同一輪 owner packet 產生後，p
v1.80 / v1.81 credential escrow intake scorecard rule：同一輪 owner response preflight 後，必須用 `scripts/reboot-recovery/post-reboot-credential-escrow-intake-scorecard.py --summary-file "$ARTIFACT_DIR/summary.txt" --owner-packet-file <owner-packets.json> --response-file <owner-response-template-or-candidate.json> --offsite-report-file <offsite-report.txt> --escrow-status-file <escrow-status.txt>` 收斂 DR escrow gate。scorecard 只讀 sanitized artifacts；不得讀 secret value、不得寫 marker、不得送 owner request、不得開 runtime gate。placeholder readback 期望 `STATUS=blocked_waiting_non_secret_credential_escrow_evidence`、`EFFECTIVE_ESCROW_MISSING_COUNT=5`、`OWNER_RESPONSE_RECEIVED_COUNT=0`、`OWNER_RESPONSE_ACCEPTED_COUNT=0`、`RUNTIME_GATE_COUNT=0`、`CREDENTIAL_MARKER_WRITE_AUTHORIZED_COUNT=0`。若未來收到合格 redacted owner response 並由 preflight 回 `ready_for_independent_reviewer_acceptance`，scorecard 應轉為 `STATUS=ready_for_independent_reviewer_acceptance`；即使 marker 尚未寫入，也只能進 `independent_reviewer_acceptance_then_marker_dry_run`，不得直接寫 marker 或宣稱 `DR_COMPLETE`。
2026-07-01 21:05 latest live summary：backup / DR freshness 已收斂，但仍不可宣稱 full green 或 10 分鐘全自動恢復完成。188 `backup_from_110` 已受控重跑成功，Harbor / Gitea / bitan backup 均 OK；188 exporter 讀回 `backup_from_110 fresh=1`、Gitea private bundle `expected_repo_missing_count=0`、`failed_repo_count=0`。110 `awoooi_db` high-frequency backup 20:00 因 `too many connections for role "awoooi"` 失敗；本輪在 DB 舊 OR 查詢為 `0`、Postgres CPU 已低的條件下走 controlled short window，產生 Restic snapshot `211a8948`，時間 `2026-07-01T20:58:20+08:00`，tags `service:awoooi,freq:6h,timestamp:20260701_204344`，tmp dump 已清除。正式保守調整為 `awoooi CONNECTION LIMIT 4`，保留 `statement_timeout=750ms`、`max_parallel_workers_per_gather=0`、`enable_seqscan=off`；rollback 是 `ALTER ROLE awoooi CONNECTION LIMIT 2` 並保留同樣 role config。110 backup exporter 讀回 `awoooi_backup_job_fresh{job="awoooi_db"} 1`、`awoooi_backup_dr_phase{next_step="complete_credential_escrow_review"} 3`、`awoooi_backup_dr_credential_escrow_missing_count 5`。`full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` 回 `PASS=93 WARN=3 BLOCKED=0`；`full-stack-recovery-scorecard.sh` 回 `CORE_COLD_START_WARN_GATES=3`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`NEXT_STEP=complete_credential_escrow_review`。Allowed declaration：public routes / AWOOOI service / Gitea / Harbor registry / 188 backup-from-110 / 110 awoooi_db freshness 已恢復，cold-start hard blockers `0`。Forbidden declaration：full green、10 分鐘全自動恢復完成、DR complete、credential escrow complete、MOMO data 最新、110 SSH 永久穩定。剩餘 WARN 是 MOMO current-month/source freshness warning 與 110 aggregate `backup-all` 上次 failed components。高頻全量 `pg_dump` 本輪耗時 `1152s` 並產生 `28G` dump，後續 P0 必須建立專用 backup DB role / `AWOOOI_BACKUP_DATABASE_URL`，並把大型可重建 snapshot 表改成分層備份或 retention 後納入 full backup；不得把 app role 放寬當長期唯一解法。
2026-07-01 19:32 latest live summary：110 卡住的根因已確認並修復。主機 ping / SSH / Prometheus / Gitea / registry 均可達，但 `awoooi-startup-110.service` 因 `/usr/local/bin/awoooi-startup-110.sh` 被替換成 runner/CD lane fail-closed stub 而固定 `exit 75`，造成 post-reboot systemd degraded 與 cold-start warning。live stub hash 為 `55e1e87d44cf5b8cefb714a49d085b886697d856569ce9478537059d49129f88`，repo 完整 startup script hash 為 `fcc67c7dde889b3cf8ffeca89cc5e58407fabc6f5a80554e39f3f1465f90b318`。修復方式固定使用 `scripts/reboot-recovery/repair-110-startup-script-stub.sh`：預設 `--verify` 只讀；`--apply` 只上傳 repo 完整 startup script、備份 live stub、解除 immutable、install 到 `/usr/local/bin/awoooi-startup-110.sh`、`systemctl daemon-reload`，並只 `reset-failed awoooi-startup-110.service`；`--start-service` 需另外明確指定，因為它會跑完整 startup checks。live apply 指令為 `BACKUP_SUFFIX=before-full-startup-restore-20260701-1928 bash scripts/reboot-recovery/repair-110-startup-script-stub.sh --apply`，post-readback `remote_matches_expected=1`、`remote_is_runner_failclosed_stub=0`，rollback backup 為 `/usr/local/bin/awoooi-startup-110.sh.before-full-startup-restore-20260701-1928`。之後清除 stale `systemd-ask-password-console.path` / `systemd-ask-password-wall.path` failed state；110 已回 `systemctl is-system-running=running`、failed units `0`。最新 cold-start 讀回 `PASS=90 WARN=6 BLOCKED=0`、`FAILED_UNITS_110 0`；110 startup stub / systemd degraded 不再是 blocker。禁止再把 live `/usr/local/bin/awoooi-startup-110.sh` 當 runner/CD lane opener 封成 stub；runner fail-closed enforcer 只能封 runner binary、runner unit、controlled opener artifact 或 upload opener source，不得封 host startup recovery script。Allowed declaration：110 不再卡在 startup stub / systemd degraded，cold-start hard blockers `0`。Forbidden declaration：full green、10 分鐘全自動恢復已完成、DR complete、credential escrow complete、MOMO data 最新或 110 SSH 永久穩定；剩餘 warnings 仍需由 backup / DR escrow / alert 收斂處理。
2026-07-01 19:13 live summary：全主機重啟後的 service hard blockers 已清零，但仍不可宣稱 DR complete 或 full green。`full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` 回 `PASS=89 WARN=7 BLOCKED=0`；public routes / TLS 含 `https://signoz.wooo.work/` 均回 expected 2xx/3xx，AWOOOI API health `200`，Gitea version API `200`，Gitea CD latest visible `#4283` 為 `Success`。MOMO dedicated source preflight 已從 hard blocker 降為 source freshness warning：`MOMO_DRIVE_TOKEN_SOURCE_PREFLIGHT PASS=19 WARN=6 BLOCKED=0`、`DB_DAILY_FRESHNESS 7|2026-06-24`、`DRIVE_INTAKE_COUNT=0`、`DRIVE_FAILED_COUNT=0`、`DRIVE_GLOBAL_LATEST_MODIFIED=2026-06-25T04:21:47Z`、latest import job `57 completed` 且 `15383|15383|0`，因此「沒有比最後乾淨 import 更新的 Drive source」不得再硬擋 cold-start；若未來 auth failure、failed folder 有候選、新 source 未匯入或 latest import 不乾淨，仍 hard block。110 cold-start monitor deploy parity 已擴充到 `full-stack-cold-start-check.sh`、`cold-start-textfile-exporter.sh` 與 `momo-drive-token-source-recovery-preflight.sh` 三者 hash；latest remote textfile metric 為 `awoooi_cold_start_pass_gates=89`、`warn=7`、`blocked=0`、`last_result{result="degraded"}=1`。SignOz 的 root-owned Nginx source drift 已在 repo 修成 `192.168.0.110:8080`，但 188 `ollama` sudo 仍需要密碼；為了不讀密碼、不 reload Nginx，本輪用 user-level controlled bridge 先恢復 live route：`scripts/reboot-recovery/signoz-188-upstream-bridge.sh --apply` 在 188 建立 `/home/ollama/bin/awoooi-signoz-upstream-bridge.sh` 與 user crontab `@reboot`，以 `socat` 監聽 `127.0.0.1:3301` 並轉到 `192.168.0.110:8080`；status readback `SIGNOZ_BRIDGE_STATUS running=1 listen_code=200 upstream_code=200`，rollback 是 `scripts/reboot-recovery/signoz-188-upstream-bridge.sh --rollback`。19:13 scorecard 讀回 `CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_WARN_GATES=7`、`CORE_COLD_START_FIRING_ALERTS=2`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`ESCROW_MISSING_COUNT=5`、`RECOVERY_STATE=CORE_NOT_READY_DR_OFFSITE_PENDING`。Allowed declaration：網站 / public routes / AWOOOI service / Gitea / Harbor registry path / SignOz public route 已恢復，cold-start hard blockers `0`。Forbidden declaration：10 分鐘全自動恢復已完成、full green、DR complete、credential escrow complete、MOMO business data 最新、110 SSH 永久穩定或 Nginx privileged source apply 已完成。下一步固定為 P0 backup/DR escrow/alert warning 收斂：credential escrow non-secret evidence 仍缺 `5`，backup health 仍有 stale / failed-component warning。