From 6250a94b7e1a3a29930070fb9a549e1803ea0a69 Mon Sep 17 00:00:00 2001 From: ogt Date: Fri, 26 Jun 2026 06:54:49 +0800 Subject: [PATCH] fix(ops): harden 188 startup data recovery gate --- docs/LOGBOOK.md | 21 ++++++++++++ docs/runbooks/FULL-STACK-COLD-START-SOP.md | 20 +++++++---- docs/runbooks/REBOOT-RECOVERY-SOP.md | 34 +++++++++++++------ ...oot-cold-start-backup-recovery-workplan.md | 2 +- scripts/reboot-recovery/awoooi-startup.sh | 16 ++++----- 5 files changed, 67 insertions(+), 26 deletions(-) diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index c04b69537..7901bfa22 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,24 @@ +## 2026-06-26｜188 host hygiene readback：服務綠燈但 systemd degraded 不可假綠 + +**背景**：全主機 readback 顯示 188 產品服務健康，但 `systemctl is-system-running=degraded`。本段只做 read-only triage 與 repo-side SOP / startup safety hardening；沒有在 188 上 reset-failed、restart、renew cert、reload Nginx、修改 DB 或讀 secret。 + +**Read-only evidence**： +- 188 failed units：`awoooi-startup.service`、`postgresql@14-main.service`、`certbot.service`、`snap.certbot.renew.service`。 +- 產品面仍綠：Docker active；MOMO / scheduler / bot healthy；VibeWork web / PostgreSQL healthy；MinIO / node-exporter / postgres-exporter / redis-exporter / nginx-exporter up；SignOz HTTP `200`；`pg_exporter_last_scrape_error=0`、`pg_up=1`、`redis_up=1`。 +- Host PostgreSQL 分層：`postgresql.service` 是 `active (exited)`，但 `pg_lsclusters` 顯示 host cluster `14/main` status `down`；`pg_isready` 與 exporter OK 不能直接替代 host cluster unit health，因為 188 也有 containerized PostgreSQL / product DB。 +- `awoooi-startup.service` 於 2026-06-23 14:50 嘗試啟動 host PostgreSQL，journal 顯示 `could not locate a valid checkpoint record`，startup 腳本隨後嘗試以 root 執行 `pg_resetwal` 但失敗。這證明原 startup 自動資料層修復邏輯不安全，不能納入專業重啟綠燈。 +- Certbot：`sentry.wooo.work` renewal 多次 `Some challenges have failed`，2026-06-26 00:05 又遇到 ACME `rateLimited / Service busy; retry later`；public certificate for `sentry.wooo.work` / shared routes still valid until `2026-07-09 16:03:40 UTC`，因此目前不是立即 route outage，但已是 P1 TLS renewal blocker。 + +**Repo-side hardening**： +- `scripts/reboot-recovery/awoooi-startup.sh` 已改為 fail-closed：遇到 PostgreSQL checkpoint/WAL 類錯誤時，只記錄需要 DB owner、backup / restore evidence、maintenance window 與 post-check；不再自動執行 `pg_resetwal`。 +- `FULL-STACK-COLD-START-SOP.md` 升到 v1.63，將 188 狀態拆成 product service green 與 host hygiene debt。 +- `REBOOT-RECOVERY-SOP.md` 將 PostgreSQL WAL 修復從自動/一般修復降級為 break-glass 手動程序，禁止當作快速重啟 SOP 的預設步驟。 + +**仍 blocked / 不得宣稱**： +- 188 `HOST_PERFECT_GREEN` 不可宣稱；目前是 `SERVICE_GREEN_HOST_HYGIENE_BLOCKED`。 +- 不得用 `pg_isready OK`、Docker DB healthy 或 public route `200` 取代 host `postgresql@14-main` 狀態。 +- 不得自動 `pg_resetwal`、`certbot renew`、`systemctl reset-failed`、Nginx reload 或 DB restore；需要 owner-approved maintenance window、rollback owner、備份 evidence 與 post-check。 + ## 2026-06-26｜AI Agents 專業執行跑道正式讀回：全域控管可見化，但仍不誇大自動接管 **背景**：統帥要求 AI Agents 不只列在治理卡片，而要能「遍地開花」控管主機、產品、網站、套件、服務、工具、AI Agent 與 AI 解決方案，並且讓使用者能看見 OpenClaw、Hermes、Nemotron 等角色的專業分工、互動、學習與成長證據。同時前端禁止顯示內部對話內容、raw path、secret 或 Telegram token。 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index ed7bb5d99..a762ab920 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,6 +1,6 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.62 +> Version: v1.63 > Last updated: 2026-06-26 Asia/Taipei > Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path. @@ -12,6 +12,8 @@ 若只是重啟後要快速判斷能不能宣稱恢復，先跑一頁式總檢查：`scripts/reboot-recovery/post-start-quick-check.sh --no-color`，並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為人工 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。 +2026-06-26 06:50-06:55 188 host hygiene read-only triage：188 product services remain green, but host `systemctl` is still `degraded` and must not be smoothed into full host green. Failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, and `snap.certbot.renew.service`. Evidence shows the host PostgreSQL cluster `14/main` is down in `pg_lsclusters`, while product DB / exporters still respond through containerized services; therefore `pg_isready` or `pg_up=1` cannot substitute for host cluster health. The 188 startup service detected `could not locate a valid checkpoint record` on 2026-06-23 and attempted `pg_resetwal` as root, which failed; v1.63 treats PostgreSQL checkpoint/WAL errors as break-glass only and the repo-side startup script now fails closed instead of running `pg_resetwal`. Certbot renew for `sentry.wooo.work` is also failing and hit ACME rate-limit / challenge failure, but the public cert is still valid until 2026-07-09 16:03:40 UTC. Current declaration: `SERVICE_GREEN_HOST_HYGIENE_BLOCKED` for 188, while overall service recovery remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`. + 2026-06-26 06:40-06:44 全主機 read-only refresh：`110 / 120 / 121 / 188 / 112 / 111 / 168` ping 與 SSH port 全部 OK。核心 reboot scope 維持 green：110 `systemctl=running`、failed units `0`，Docker / Gitea / Harbor / Prometheus / Alertmanager 可用；120 / 121 `systemctl=running`、failed units `0`，K3s nodes `mon` / `mon1` Ready；188 產品容器與 PostgreSQL / Redis / MOMO / SignOz 可用。ArgoCD `awoooi-prod` 已從先前 degraded 收斂為 `Synced / Healthy`，revision `b2945ab9f716d9d685434ae0e67b9318414b27fe`；`km-vectorize` official 03:00 台北時間 run 成功，`lastSuccess=2026-06-25T19:00:14Z`。Public routes for AWOOOI / VibeWork / AwoooGo / MOMO / Stock / Bitan / Gitea / Harbor / Registry / Sentry / SigNoz / Langfuse return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health is `V10.690`; StockPlatform freshness is `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status remains core green with `escrow_missing=5`. Boundaries: 188 host still has failed units `awoooi-startup.service`, `certbot.service`, `postgresql@14-main.service`, `snap.certbot.renew.service` that require host hygiene cleanup; 112 Wazuh services / ports are active but Wazuh manager registry accepted remains `0`; 111 / 168 Codex workspaces are reachable but have different local HEADs on the same ahead branch; Mac Mini free space is about `3.4Gi`. Current service verdict remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DR_COMPLETE` or `Wazuh recovered`. 2026-06-26 06:26-06:28 隔日 read-only refresh：四主機 ping/SSH OK，cold-start `PASS=89 WARN=0 BLOCKED=0`，MOMO `V10.690` 且 latest import job `57 completed`，StockPlatform `/api/v1/system/freshness` 仍為 `status=ok` / `latest_trading_date=2026-06-25` / blockers `[]`，backup-status 110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`last_backup_all=2026-06-26 02:31:02`、`escrow_missing=5`。06:26 full wrapper 首輪在 `https://awoooi.wooo.work/zh-TW/iwooos` 與 `https://vibework.wooo.work/` 出現單次 `000`，但獨立 curl 立即回 `200`，route-only wrapper 也回 `PASS=31 WARN=0 BLOCKED=0 RESULT=GREEN`；因此 v1.61 將 public route gate 改為最多 3 次 retry，只有連續失敗才算 `BLOCKED`，retry 後恢復則列為 evidence warning。06:28 core wrapper with routes skipped returns `POST_START_QUICK_CHECK PASS=15 WARN=2 BLOCKED=0`, `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。本次沒有 Docker/systemd/Nginx/firewall/K8s/DB/Wazuh runtime 寫操作。 @@ -134,7 +136,7 @@ Forbidden declaration: AI runtime remediation is enabled. Process termination, D | 110 host runaway process guard | 14:31-14:32 live scrape confirms `monitor_up=1`, orphan browser groups `0`, active Gitea Actions containers `2`, `load5_per_core≈0.79-0.81`, `swap_used_ratio≈1.0`, and `remediation_authorized=0`; exporter/helper also remain in Ansible textfile exporter source-of-truth. | `LIVE_SCRAPED_RUNTIME_GATE_0` | | 120 reachability | ping OK, SSH OK, boot around `2026-06-14 02:23`, K3s active, node `mon Ready` | `GREEN` | | 121 reachability | ping OK, SSH OK, failed units `0` | `GREEN` | -| 188 host runtime | production services green; host degraded only by `certbot.service` and `snap.certbot.renew.service` | `GREEN_WITH_CERTBOT_DEBT` | +| 188 host runtime | production services green, but host `systemctl` degraded by `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, and `snap.certbot.renew.service`; host PostgreSQL cluster `14/main` is down while product DB containers/exporters are healthy; certbot renewal for shared `sentry.wooo.work` certificate is failing but public cert is still valid until 2026-07-09 UTC | `SERVICE_GREEN_HOST_HYGIENE_BLOCKED` | | K3s node state | `mon Ready control-plane`, `mon1 Ready control-plane`; bad pods `0`; `FAILED_JOBS=1`, `STALE_FAILED_JOBS=1`, `ACTIVE_FAILED_JOBS=0` | `GREEN_WITH_RETAINED_EVIDENCE` | | 110 -> 120 / 188 SSH trust | 00:33 cold-start exposed stale `known_hosts`; backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; final repair backup `/home/wooo/.ssh/known_hosts.before-120-188-final-refresh.20260613-011949`; CD fix `80e6ec1a` moves deploy trust to `/home/wooo/.ssh/deploy_known_hosts`; 01:28 global `known_hosts` still contains 120 / 188 and was not clobbered by deploy marker `e4a349bc` | `GREEN_WITH_GUARDRAIL` | | Backup status | 11:20 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`; escrow readback still shows `ESCROW_MISSING_COUNT=5` | `GREEN_WITH_DR_ESCROW_WARNING` | @@ -591,11 +593,13 @@ ssh ollama@192.168.0.188 'hostname; date; who -b; uptime; systemctl is-system-ru 2026-06-12 110 事故教訓：failed unit 若指向不存在的 legacy 路徑，要先確認是否仍屬現役服務。停用 stale timer 可以解除 host degraded，但必須同步 source-of-truth 後續清理，不能靠反覆 `reset-failed` 掩蓋。 +2026-06-26 188 事故教訓：PostgreSQL host cluster / Docker product DB / exporter 三者必須分開判定。`pg_isready`、`pg_up=1` 或 public route `200` 只能證明某個 PostgreSQL endpoint 可用，不能證明 `postgresql@14-main` 已恢復。若 journal 出現 `could not locate a valid checkpoint record`，不得由 startup 腳本或 AI 自動執行 `pg_resetwal`；必須進入 DB owner / backup restore / maintenance window / rollback owner / post-check gate。 + --- ## 4. P0 188 Data Layer -188 is the first real service dependency because K3s datastore and AWOOOI DB depend on PostgreSQL. +188 is the first real service dependency because multiple product data planes, exporters, and AI / observability services depend on PostgreSQL-compatible endpoints. Do not assume the host cluster `postgresql@14-main`, Docker product databases, and exporter target are the same endpoint; prove the authoritative endpoint before repair. ### 4.1 Startup order @@ -616,6 +620,8 @@ ssh ollama@192.168.0.188 'hostname; date; who -b; uptime; systemctl is-system-ru ssh ollama@192.168.0.188 ' hostname; date; uptime; free -h systemctl is-active containerd docker postgresql@14-main redis-server ollama nginx || true +pg_lsclusters 2>/dev/null || true +ss -ltnp "sport = :5432" 2>/dev/null || ss -ltn "sport = :5432" || true pg_isready -h localhost -p 5432 || true redis-cli -p 6380 ping 2>/dev/null || redis-cli ping 2>/dev/null || true docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | head -120 @@ -632,14 +638,16 @@ invalid primary checkpoint record unexpected pageaddr ... in log segment ... ``` -This blocks: +This may block if the affected cluster is the authoritative runtime datastore: - `188:5432` - K3s startup on 120/121 - AWOOOI API DB access - Alertmanager webhook if API cannot start -Human-approved recovery command on 188: +2026-06-26 counterexample: host cluster `14/main` can be down while product DB containers and exporters still serve traffic. Therefore `pg_isready` is not enough and failed `postgresql@14-main` is not automatically a product outage. First map the listening process / container, current app DB configuration, and backup freshness. + +Break-glass example only after DB owner approval, backup evidence, maintenance window, rollback owner, and post-check plan: ```bash sudo systemctl stop postgresql@14-main @@ -651,7 +659,7 @@ pg_isready -h localhost -p 5432 sudo -u postgres psql -d k3s_datastore -c "VACUUM ANALYZE kine;" ``` -Do not run `DROP`, reinitialize the cluster, delete `/var/lib/postgresql`, or restore an old backup unless the commander explicitly approves it. +Do not run `pg_resetwal`, `DROP`, reinitialize the cluster, delete `/var/lib/postgresql`, or restore an old backup from AI/startup automation. These are break-glass actions only. --- diff --git a/docs/runbooks/REBOOT-RECOVERY-SOP.md b/docs/runbooks/REBOOT-RECOVERY-SOP.md index ad677526d..3671ea52c 100644 --- a/docs/runbooks/REBOOT-RECOVERY-SOP.md +++ b/docs/runbooks/REBOOT-RECOVERY-SOP.md @@ -157,7 +157,7 @@ cat /var/log/harbor-watchdog.log # 持久化 log **188 (7 步驟)**: - containerd BoltDB 損壞偵測與修復 - Docker BoltDB 損壞偵測與修復 -- PostgreSQL WAL 損壞偵測 + pg_resetwal + kine VACUUM +- PostgreSQL 啟動偵測；checkpoint / WAL 類資料層錯誤一律 fail-closed，禁止自動 `pg_resetwal` - Redis 啟動 - Ollama 啟動 - Nginx 啟動 @@ -316,15 +316,25 @@ BOLT=/var/lib/docker/network/files/local-kv.db cp $BOLT ${BOLT}.bak.$(date +%s) && rm $BOLT && systemctl restart docker ``` -**PostgreSQL WAL 修復**: -```bash -# 確認是 WAL 損壞 -journalctl -u postgresql@14-main -n 30 | grep "could not locate a valid checkpoint" +**PostgreSQL checkpoint / WAL 類錯誤處理**: -# 修復 -systemctl stop postgresql@14-main -/usr/lib/postgresql/14/bin/pg_resetwal -f /var/lib/postgresql/14/main -systemctl start postgresql@14-main +> 2026-06-26 校正：這不是一般重啟 SOP 的自動修復步驟。`pg_resetwal` 可能造成不可逆資料風險，只能在 DB owner、備份/restore evidence、maintenance window、rollback owner 與 post-check plan 都具備時，作為 break-glass 人工程序。AI、startup 腳本或值班腳本不得自動執行。 + +```bash +# 只讀確認，不代表可執行修復 +journalctl -u postgresql@14-main -n 30 | grep "could not locate a valid checkpoint" +pg_lsclusters +systemctl status postgresql@14-main --no-pager +``` + +若確認需要進入 break-glass，先完成： + +```text +1. DB owner approval +2. 最新本機與 offsite backup evidence +3. restore / rollback plan +4. maintenance window +5. post-check：pg_lsclusters、pg_isready、應用 DB smoke、backup-status、cold-start wrapper ``` ### 192.168.0.110 — DevOps 主機 @@ -500,7 +510,9 @@ curl http://192.168.0.110:9090/api/v1/query?query=probe_success - Redis: dump.rdb 損壞 (`Wrong signature trying to load DB from file`) - ClickHouse: system table store parts 損壞 (`broken and needs manual correction`) -**修復步驟**: +**Break-glass 修復範例，不屬於一般重啟 SOP**: + +先確認 Sentry DB volume 是受影響權威資料面，並具備 owner approval、volume backup / rollback ref、maintenance window 與 post-check plan。不得由 startup script、AI agent 或值班自動流程直接執行。 ```bash # Step 1: 修復 PostgreSQL WAL @@ -533,7 +545,7 @@ curl -o /dev/null -w "%{http_code}" http://192.168.0.110:9000/ # 預期: 200 / 302 / 400 (需登入) ``` -**startup-110.sh 已自動化修復**: PostgreSQL WAL 和 Redis RDB 損壞會自動偵測並修復。ClickHouse parts 損壞需手動識別 UUID（因 UUID 每次不同）。 +**2026-06-26 修正方向**: 舊 `startup-110.sh` 曾把 PostgreSQL WAL 和 Redis RDB 損壞列入自動修復，這類資料層破壞性動作後續必須遷移到 owner-gated break-glass，不得再擴大成預設自動化。ClickHouse parts 損壞同樣需人工識別 UUID、備份與 post-check。 --- diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index 8eac7aa6a..87cead325 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -11,7 +11,7 @@ | Area | Status | Completion | Evidence | |------|--------|------------|----------| -| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-26 06:40-06:44 all-host read-only refresh confirms the 2026-06-25 21:14 green baseline still holds and adds 112/111/168 reachability. `110 / 120 / 121 / 188 / 112 / 111 / 168` ping and SSH port are OK. Core scope services are green: cold-start `PASS=89 WARN=0 BLOCKED=0`; public routes return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health `V10.690`, latest import job `57 completed`, daily freshness `1|2026-06-24`; StockPlatform freshness `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, offsite/rclone fresh, `last_backup_all=2026-06-26 02:31:02`, `escrow_missing=5`. Do not declare DR complete until `escrow_missing=0`; Wazuh manager registry accepted remains `0`; 111/168 Codex workspace HEAD drift and Mac Mini low free space are workstation blockers, not reboot service blockers. | +| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-26 06:40-06:55 all-host read-only refresh confirms the 2026-06-25 21:14 green baseline still holds and adds 112/111/168 reachability. `110 / 120 / 121 / 188 / 112 / 111 / 168` ping and SSH port are OK. Core scope services are green: cold-start `PASS=89 WARN=0 BLOCKED=0`; public routes return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health `V10.690`, latest import job `57 completed`, daily freshness `1|2026-06-24`; StockPlatform freshness `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, offsite/rclone fresh, `last_backup_all=2026-06-26 02:31:02`, `escrow_missing=5`. 188 is now explicitly split into product-service green and host hygiene blocker: failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, `snap.certbot.renew.service`; host PostgreSQL cluster is down while product DB containers/exporters are up; certbot renewal is failing but public cert remains valid until 2026-07-09 UTC. Do not declare DR complete until `escrow_missing=0`; Wazuh manager registry accepted remains `0`; 111/168 Codex workspace HEAD drift and Mac Mini low free space are workstation blockers, not reboot service blockers. | | P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-26 06:44 readback shows 120 and 121 reachable, K3s active, `mon` and `mon1` both `Ready control-plane`, AWOOOI API/Web replicas split across both nodes, ArgoCD `awoooi-prod Synced / Healthy` at revision `b2945ab9f716d9d685434ae0e67b9318414b27fe`, and `km-vectorize` official 03:00 台北時間 run succeeded with `lastSuccess=2026-06-25T19:00:14Z`. | | P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-25 19:17 backup readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-25 02:35:09`。2026-06-25 19:19 offsite escrow report shows script presence OK, rclone configured, full and partial rclone markers present, `PASS=8 WARN=5 BLOCKED=0`, `ESCROW_MISSING_COUNT=5`; DR remains blocked on real non-secret credential escrow evidence IDs. | | P2 service / data truth | DONE | 100% | Service routes and core runtime are available, 110 orphan Chrome CPU pressure is cleared, and StockPlatform cron-source drift is repaired. 2026-06-25 21:13 StockPlatform `/api/v1/system/freshness` returned `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; price / chips / margin / AI recommendations are all on `2026-06-25`. `ai.recommendations` row count is `2868`; `core.margin_short_daily` row count is `1976`. MOMO health `V10.690`, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `DB_DAILY_FRESHNESS 1|2026-06-24` are green; expanded public routes are green. | diff --git a/scripts/reboot-recovery/awoooi-startup.sh b/scripts/reboot-recovery/awoooi-startup.sh index 6952bfc27..8a91db2f1 100644 --- a/scripts/reboot-recovery/awoooi-startup.sh +++ b/scripts/reboot-recovery/awoooi-startup.sh @@ -1,6 +1,8 @@ #!/bin/bash # AWOOOI 重開機自動恢復腳本 -# 2026-04-04 ogt: 根據實際事故建立，解決 PostgreSQL WAL 損壞 + Docker BoltDB 損壞 + K3s Kine 慢查詢 +# 2026-04-04 ogt: 根據實際事故建立，處理 container / Docker 啟動順序與 K3s Kine 維護。 +# 2026-06-26 Codex: PostgreSQL checkpoint/WAL 錯誤改為 fail-closed； +# 不在自動啟動腳本內執行 pg_resetwal，避免資料破壞被誤判成恢復。 # 部署位置: /usr/local/bin/awoooi-startup.sh (on 192.168.0.188) # systemd unit: /etc/systemd/system/awoooi-startup.service @@ -67,7 +69,7 @@ else fi # ────────────────────────────────────────────── -# STEP 3: PostgreSQL 修復（若損壞） +# STEP 3: PostgreSQL 啟動檢查（資料層錯誤必須人工 gate） # ────────────────────────────────────────────── log "[3/7] 檢查 PostgreSQL..." @@ -78,13 +80,11 @@ if ! systemctl is-active postgresql@14-main >/dev/null 2>&1; then fi if ! systemctl is-active postgresql@14-main >/dev/null 2>&1; then - log "PostgreSQL 啟動失敗，檢查是否 WAL 損壞..." + log "PostgreSQL 啟動失敗，檢查是否屬於 checkpoint/WAL 類資料層錯誤..." if journalctl -u postgresql@14-main -n 20 | grep -q "could not locate a valid checkpoint"; then - log "⚠️ WAL 損壞！執行 pg_resetwal..." - /usr/lib/postgresql/14/bin/pg_resetwal -f /var/lib/postgresql/14/main - log "WAL 重置完成，重啟 PostgreSQL..." - systemctl start postgresql@14-main - sleep 8 + log "❌ 偵測到 PostgreSQL checkpoint/WAL 錯誤；禁止自動 pg_resetwal。" + log "需要 DB owner、備份/restore evidence、maintenance window 與 post-check 後才能人工處理。" + exit 1 fi systemctl is-active postgresql@14-main && log "✅ PostgreSQL 修復成功" || { log "❌ PostgreSQL 修復失敗"; exit 1; } fi