wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 4 Packages Projects Releases Wiki Activity
Files
ab242018b193cf252b1593fe73ca8a66dcbadc4e
awoooi/docs/runbooks/FULL-STACK-COLD-START-SOP.md

59 KiB
Raw Blame History

AWOOOI 全棧冷啟動與主機重啟 SOP

Version: v1.9 Last updated: 2026-06-13 Asia/Taipei Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.

0. 最新 Live Baseline 與釋出判定

本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md。

項目 2026-06-13 14:16 Asia/Taipei live result 判定
Overall recovery readiness 96% SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED
P0 host / K3s recovery 100% DONE
P1 backup / alert / escrow 92% BLOCKED_DR_ESCROW
P2 service / data truth 100% VERIFIED_WORKLOAD_BALANCED
P3 docs / automation contracts 100% DONE_WITH_VALIDATION_GAP
110 host runtime systemctl is-system-running=running, failed units 0, cold-start load 3.55 / 4.54 / 4.51; Docker / Harbor / Gitea / Prometheus / Alertmanager / Sentry checks pass GREEN_WITH_LOAD_WATCH
120 reachability ping OK, SSH OK, boot 2026-06-12 15:13, root ext4 rw, failed units 0 GREEN
121 reachability ping OK, SSH OK, failed units 0 GREEN
188 host runtime production services green; host degraded only by certbot.service and snap.certbot.renew.service GREEN_WITH_CERTBOT_DEBT
K3s node state mon Ready control-plane, mon1 Ready control-plane, no failed jobs / bad pods; latest CD kept API/Web split across 120 / 121 GREEN_WORKLOAD_BALANCED
110 -> 120 / 188 SSH trust 00:33 cold-start exposed stale known_hosts; backup /home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416; final repair backup /home/wooo/.ssh/known_hosts.before-120-188-final-refresh.20260613-011949; CD fix 80e6ec1a moves deploy trust to /home/wooo/.ssh/deploy_known_hosts; 01:28 global known_hosts still contains 120 / 188 and was not clobbered by deploy marker e4a349bc GREEN_WITH_GUARDRAIL
Backup status 14:16 status: 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, escrow_missing=5, last aggregate 2026-06-13 02:34:06 GREEN_WITH_DR_ESCROW_WARNING
Offsite sync / verify 01:28 textfile: awoooi_backup_offsite_remote_verify_ok=1, full_verify_fresh=1, all 13 repos have snapshot_count=1 and snapshot_latest_only=1; latest scheduled verifier log is 2026-06-12 07:20 GREEN
Backup / cold-start alerts 01:27 live visibility check confirms Prometheus and Alertmanager expose the 5 required credential escrow gap alerts; Prometheus rules API has all five required alert names healthy; label contract check loads 24 baseline backup alert rules GREEN_WITH_EXPECTED_REDLIGHTS
Cold-start scorecard 14:16 read-only scorecard after production image e897c8bf verification: PASS=83 WARN=0 BLOCKED=0 GREEN
momo DB parity `4571 4571
momo scheduler container healthy; scorecard reads SCHEDULER_RECENT_ACTIVITY 1136; detector widened and deployed to 110 GREEN
ArgoCD app health 14:16 ArgoCD revision a520c32d is Synced; app remains Degraded only because km-vectorize CronJob has not completed its last execution successfully. CronJob schedule is 0 3 * * * with timeZone=Asia/Taipei, failedJobsHistoryLimit=3, and lastSuccessfulTime=2026-06-04T11:00:37Z; next 03:00 run must prove success or leave inspectable failure evidence. GOVERNANCE_DEBT_IN_REMEDIATION
Workload balancing Live API/Web/Worker image is e897c8bf; API/Web pods remain split across 120 / 121, Worker single replica remains healthy GREEN
Credential escrow 5 non-secret evidence markers missing BLOCKED

Release rule:

Do not declare full cold-start green unless the latest scorecard has `WARN=0` and `BLOCKED=0`.
Do not declare aggregate backup green unless latest `backup-status` has `core_blockers=0`.
Do not declare DR scorecard complete while credential escrow markers are missing.

2026-06-13 live rule:

110 / 120 / 121 / 188 service recovery is full-stack green after the 14:16 scorecard.
GO for controlled runner/CD release; keep AI auto-remediation governed by normal gates.
NO-GO for "DR complete" while credential escrow evidence markers are missing.
Do not fake or silence credential escrow alerts; they are the remaining correct DR red light.
GO for "AWOOOI core workload balanced"; topology spread is in Gitea main / ArgoCD live and API/Web placement proves max skew <= 1.
NO-GO for "ArgoCD fully healthy" until `km-vectorize` updates `lastSuccessfulTime` after an official scheduled Job, not a manual `UnexpectedJob`.
NO-GO for any CD workflow that writes deploy host keys into `/home/wooo/.ssh/known_hosts`; deploy jobs must use an isolated `UserKnownHostsFile=/home/wooo/.ssh/deploy_known_hosts`.
Current allowed wording: "service / cold-start green; DR complete still blocked by credential escrow; ArgoCD CronJob governance debt in remediation; `km-vectorize` failure evidence retention is fixed and the next 03:00 run must prove success."

After any future 120 recovery, rerun this exact chain from 110:

/backup/scripts/backup-configs.sh
/backup/scripts/backup-all.sh
/backup/scripts/sync-offsite-backups.sh --mode sync
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
/home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1

0.1 When To Use This

Use this SOP when any of these happen:

  • 110/120/121/188 reboot unexpectedly.
  • All services are abnormal after a power/network event.
  • K3s is stuck activating.
  • Host load remains high during startup and service health is mixed.
  • Monitoring, alerting, CD, AI auto-repair, and Docker Compose services disagree about the real state.

The rule is simple: recover the dependency chain, not the loudest symptom.

0.2 啟動判定分層

重啟後不能只用一個訊號宣稱完成。每台主機與整個平台都必須分四層判定：

層級 代表意義 最低證據 不代表
HOST_POWERED 主機或 VM 看起來已通電 console / hypervisor 顯示 running，或 LAN ARP 開始出現 OS 已完成開機
HOST_BOOTED OS 已進入可互動狀態 ping OK、SSH port open、who -b 有本次 boot time systemd / Docker / K3s 已健康
HOST_READY 主機基礎服務可承接下一層 systemctl is-system-running 非 degraded；failed units 可解釋；cron / docker / DB / K3s 依角色正常 public route 或業務資料已正常
SERVICE_READY 主機承載服務可用 服務 health、port、container health、DB / Redis / K3s / Harbor / Alertmanager checks 通過 備份、排程、告警、資料一致性已驗證
FULL_STACK_GREEN 可以宣稱重啟恢復完成 cold-start scorecard WARN=0、BLOCKED=0，備份/offsite/DB/告警/排程都綠 120 不可達時永遠不能宣稱

2026-06-12 的 110/120 事故收斂判定是：

110 HOST_READY = yes
120 HOST_READY = yes
Core public services SERVICE_READY = yes
FULL_STACK_GREEN = yes, because cold-start scorecard is PASS=83 WARN=0 BLOCKED=0
DR_COMPLETE = no, because credential escrow evidence is incomplete

所有回報必須使用這組詞，避免把「服務面可用」誤報成「整體 DR 完成」。

1. Golden Startup Order

0. Freeze automation and preserve evidence
1. Physical/network layer
2. 188 data layer
3. 110 registry/observability layer
4. 120/121 K3s layer
5. AWOOOI workload layer
6. Public routes and alert chain
7. High-load batch/consumer/crawler services
8. Runner/CD
9. AI auto-remediation
10. 112 Kali scanner, if needed

Never start runner/CD before 188 PostgreSQL, 110 Harbor, K3s nodes, and AWOOOI API are healthy.

1.1 Dependency Graph

flowchart TD
  network["P0 network: LAN, ARP, SSH"] --> data188["188 data: PostgreSQL, Redis, momo DB, SignOz"]
  network --> obs110["110 registry/observability: Harbor, Gitea, Prometheus, Alertmanager, Sentry"]
  data188 --> k3s["120/121 K3s: server, agent, VIP, NodePorts"]
  obs110 --> k3s
  k3s --> workload["AWOOOI workload: API, Web, K8s Secrets"]
  workload --> alertchain["Alert chain: Alertmanager webhook, Telegram"]
  workload --> public["Public routes: awoooi.wooo.work, mo.wooo.work"]
  public --> schedules["Schedules: cron, CronJobs, backups, exporters"]
  schedules --> highload["High-load release: crawlers, Snuba, ClickHouse merges, runners/CD"]
  highload --> ai["AI auto-remediation: limited execution"]

This is also captured in the machine-readable baseline:

ops/reboot-recovery/full-stack-cold-start-baseline.yml

The YAML baseline is the source of truth for:

  • hosts, roles, and SSH users
  • phase ordering
  • service startup dependencies
  • endpoint success codes
  • schedule freshness thresholds
  • stateful-service protection boundaries
  • AI automation release gates

1.2 Phase Gate Logic

Each phase has the same decision rule:

Result Meaning Action
BLOCKED A dependency required by later phases is down. Stop phase release and fix the first blocked gate.
WARN Core dependency passed, but confidence is incomplete. Continue diagnosis, but do not release runner/CD/AI full execution.
GREEN All checks in scope passed. Release the next phase only.

The cold-start flow is intentionally conservative:

P0 network green
  -> P0 188 data green
  -> P0 110 registry/observability green
  -> P1 K3s green
  -> P2 workload + alert chain green
  -> P2 public routes green
  -> P2 schedules green
  -> P3 high-load services and runners/CD
  -> AI auto-remediation limited execution

The final release condition is not "containers are running". It is:

PASS > 0
WARN = 0
BLOCKED = 0
Result: GREEN

1.3 重啟 GO / NO-GO 決策樹

每次維護前先用這張表決定是否可以重啟，以及重啟後可以宣稱到哪個層級。

情境 GO / NO-GO 可做範圍 完成宣告上限
03:00 offsite sync 正在跑 NO-GO 只讀觀察，等待 sync 結束後 verifier 不宣告維護完成
120 不可達，但只重啟 110 CONDITIONAL GO 只可宣稱 110 / public service recovery；不可跑 120 backup fix SERVICE_READY，不可 FULL_STACK_GREEN
188 data layer 不健康 NO-GO 先修 PostgreSQL / Redis / Docker / SignOz / momo DB 不釋出 K3s / runner / AI
110 Harbor / Registry 不健康 NO-GO for K3s deploy 先修 registry；K3s 可能 image pull 失敗 不釋出 CD / deploy
120 / 121 都 Ready，offsite verifier 綠 GO 可做完整 cold-start release chain 需 scorecard WARN=0 BLOCKED=0
credential escrow marker 缺失 GO for service reboot，NO-GO for DR complete 可恢復服務；不可宣稱 DR scorecard complete SERVICE_READY 或 BLOCKED_DR_ESCROW
Alertmanager required rules 不可見 NO-GO for unattended window 先修 alert rules / drift guard 不釋出 AI auto-remediation

GO 只代表允許執行指定範圍，不代表完成。完成一定要回到 §15 Done Criteria。

2. Automation Freeze

Cold start creates noisy metrics and partial failures. During P0/P1, keep automation in observe-only mode.

Item Cold-start policy Reason
Gitea/GitHub runners Last Build jobs can saturate 110 CPU/RAM.
momo-scheduler / crawlers Last Chrome and batch work can saturate 188.
Sentry/Snuba consumers Controlled Kafka backlog and ClickHouse merge can create temporary high load.
Alertmanager outbound notification Gate Avoid alert storms before API webhook and Telegram are verified.
AI auto-repair Observe-only Metrics, Redis, KM, and playbooks may be incomplete.
Stateful DB restart Human approval PostgreSQL, Redis, ClickHouse, Harbor DB, Sentry DB are not generic restart targets.

2.1 Freeze 執行清單

進入維護窗口後，先把「會放大事故」的來源降到 observe-only 或延後釋出。若沒有做到這一步，後續負載和告警會很難判讀。

順序 對象 只讀確認 允許動作 禁止動作
1 runner / CD systemctl list-units "actions.runner.*"、Gitea Actions running jobs 暫停新 job、等待可完成 job 結束 重啟 Docker daemon 來中斷 job
2 AI auto-remediation Prometheus / Alertmanager / cold-start monitor 狀態 切 observe-only、保留告警 自動 restart stateful service
3 momo scheduler / crawler container health、recent logs、DB parity 延後 heavy import、保留現有資料 在 DB 未綠時強行匯入
4 Sentry / Snuba ClickHouse / Kafka health、consumer restart loop 控制 consumer 釋出順序 generic compose down/up 全套重啟
5 K3s workload node readiness、pods、events 依 node 狀態 cordon/drain 120 不可達時宣稱 drain 成功

多個工作視窗同時處理事故時，第一優先是避免互相打斷：只要有人在收斂 Docker / Nginx / firewall / K3s 寫操作，其他視窗先只讀觀察，直到明確交接。

2.2 CD / SSH Trust Guardrail

2026-06-13 的冷啟動假紅燈顯示：CD workflow 若用 ssh-keyscan ... > /home/wooo/.ssh/known_hosts，會覆蓋 110 使用者層的全域 SSH trust，導致 110 到 120 / 188 的 strict SSH 檢查失敗。這會把實際已恢復的主機誤判成 blocked。

固定規則：

項目 正確做法 禁止
Deploy 專用 host key 寫入 /home/wooo/.ssh/deploy_known_hosts 寫入或覆蓋 /home/wooo/.ssh/known_hosts
Deploy SSH options -o UserKnownHostsFile=/home/wooo/.ssh/deploy_known_hosts 共用 operator / cold-start 的 known_hosts
冷啟動 SSH trust 保留 120 / 188 的已驗證 fingerprint；修復前先備份 無 fingerprint 交叉驗證就 ssh-keygen -R 或重建全檔
驗證 CD 後檢查 known_hosts mtime、120 / 188 entries、strict SSH 只看 CD success badge

2026-06-13 修復錨點：

  • Source fix：Gitea main 包含 80e6ec1a fix(ci): avoid clobbering runner known hosts。
  • Deploy marker：e4a349bc chore(cd): deploy 414413a [skip ci] 後，/home/wooo/.ssh/known_hosts mtime 仍停在 2026-06-13 01:20:02 +0800，未被 CD 覆蓋。
  • Deploy isolated file：/home/wooo/.ssh/deploy_known_hosts mtime 2026-06-13 01:24:05 +0800。
  • Global strict entries：120 ED25519 line 4、188 ED25519 line 5 仍存在；strict SSH 到 wooo@192.168.0.120 與 ollama@192.168.0.188 必須通過。

3. P0 Evidence And Network

Run from any machine on the same LAN:

for h in 110 120 121 188; do
  ping -c 2 -W 2 192.168.0.$h >/dev/null && echo "PING_OK 192.168.0.$h" || echo "PING_FAIL 192.168.0.$h"
done

arp -an | grep -E '192\.168\.0\.(110|120|121|188)'
for h in 110 120 121 188; do
  nc -G 3 -z 192.168.0.$h 22 && echo "SSH_OK 192.168.0.$h" || echo "SSH_FAIL 192.168.0.$h"
done

Then capture reboot evidence:

ssh ollama@192.168.0.188 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
ssh wooo@192.168.0.110 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
ssh wooo@192.168.0.120 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
ssh wooo@192.168.0.121 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'

If any host has ARP incomplete or SSH port down, stop here and fix physical/network first.

3.1 主機已啟動判定標準

每台主機重啟後先跑「四段式啟動判定」。只有全部符合角色期望，才進入服務恢復。

for h in 110 120 121 188; do
  ip="192.168.0.$h"
  echo "=== $ip ==="
  ping -c 2 -W 2 "$ip" >/dev/null && echo "HOST_POWERED_OR_LAN_OK=1" || echo "HOST_POWERED_OR_LAN_OK=0"
  arp -an | grep "$ip" || true
  nc -G 3 -z "$ip" 22 && echo "SSH_PORT_OPEN=1" || echo "SSH_PORT_OPEN=0"
done

可 SSH 後：

ssh wooo@192.168.0.110 'hostname; date; who -b; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true; free -h; swapon --show'
ssh wooo@192.168.0.121 'hostname; date; who -b; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true'
ssh ollama@192.168.0.188 'hostname; date; who -b; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true; free -h'

120 若不可 SSH，狀態只能是 HOST_POWERED_UNKNOWN 或 HOST_BOOTED_UNKNOWN，必須走 console / VM / network 檢查，不可用 Kubernetes stale node object 代替主機現況。

判定 必要條件 下一步
HOST_BOOTED ping 或 ARP 有回應、SSH port open、who -b 顯示本次 boot time 檢查角色服務
HOST_READY systemctl is-system-running 為 running，或 degraded units 已逐一解釋且不影響本 host 角色 進入服務層驗證
HOST_DEGRADED failed units 存在且影響本 host 角色，或 swap 滿、root readonly、boot storage error 先修 host，不釋出下一層
HOST_UNREACHABLE ping/SSH/ARP 失敗 停止遠端修復假設，改 console/VM/network

2026-06-12 110 事故教訓：failed unit 若指向不存在的 legacy 路徑，要先確認是否仍屬現役服務。停用 stale timer 可以解除 host degraded，但必須同步 source-of-truth 後續清理，不能靠反覆 reset-failed 掩蓋。

4. P0 188 Data Layer

188 is the first real service dependency because K3s datastore and AWOOOI DB depend on PostgreSQL.

4.1 Startup order

  1. containerd
  2. docker
  3. postgresql@14-main
  4. k3s_datastore.kine maintenance
  5. redis-server on 6380
  6. ollama or current AI proxy dependencies
  7. nginx
  8. Docker networks
  9. MinIO / OpenClaw / SignOz
  10. momo / litellm / batch services after load is stable

4.2 Read-only check

ssh ollama@192.168.0.188 '
hostname; date; uptime; free -h
systemctl is-active containerd docker postgresql@14-main redis-server ollama nginx || true
pg_isready -h localhost -p 5432 || true
redis-cli -p 6380 ping 2>/dev/null || redis-cli ping 2>/dev/null || true
docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | head -120
'

4.3 PostgreSQL WAL checkpoint damage

Signature:

PANIC: could not locate a valid checkpoint record
invalid primary checkpoint record
unexpected pageaddr ... in log segment ...

This blocks:

  • 188:5432
  • K3s startup on 120/121
  • AWOOOI API DB access
  • Alertmanager webhook if API cannot start

Human-approved recovery command on 188:

sudo systemctl stop postgresql@14-main
sudo install -d -m 700 -o postgres -g postgres /var/backups/postgresql
sudo tar -C /var/lib/postgresql/14 -czf /var/backups/postgresql/14-main-before-pg-resetwal-$(date +%Y%m%d-%H%M%S).tgz main
sudo -u postgres /usr/lib/postgresql/14/bin/pg_resetwal -f /var/lib/postgresql/14/main
sudo systemctl start postgresql@14-main
pg_isready -h localhost -p 5432
sudo -u postgres psql -d k3s_datastore -c "VACUUM ANALYZE kine;"

Do not run DROP, reinitialize the cluster, delete /var/lib/postgresql, or restore an old backup unless the commander explicitly approves it.

5. P0/P1 110 Registry And Observability

110 must recover Harbor/Gitea/Monitoring early, but runners last.

5.1 Startup order

  1. docker
  2. Remove Exited (128) / Exited (137) orphan containers
  3. Harbor harbor-log
  4. Harbor full stack
  5. Gitea
  6. Prometheus / Alertmanager / Grafana / exporters
  7. Langfuse
  8. SignOz
  9. Sentry DB layer
  10. Sentry web/worker/consumer layer
  11. Gitea host runner and actions runners

5.2 Checks

ssh wooo@192.168.0.110 '
hostname; date; uptime; free -h
systemctl is-active docker || true
curl -s -o /dev/null -w "harbor=%{http_code}\n" --max-time 5 http://127.0.0.1:5000/v2/ || true
curl -s -o /dev/null -w "gitea=%{http_code}\n" --max-time 5 http://127.0.0.1:3001/ || true
curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true
curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true
curl -s -o /dev/null -w "sentry=%{http_code}\n" --max-time 10 http://127.0.0.1:9000/ || true
docker ps --format "{{.Names}}\t{{.Status}}" | head -120
'

Harbor healthy means /v2/ returns 200 or 401. Do not treat 401 as failure.

5.3 Runner gate

Runner may start only after all are true:

  • 188 PostgreSQL ready
  • 110 Harbor ready
  • 110 Gitea ready
  • 120/121 K3s nodes ready
  • AWOOOI API health passes
  • 110 load/core is below 1.0 for at least 15 minutes
  • runner systemd guardrails are active: CPUQuota=200%, MemoryMax=2G, WatchdogUSec=0

Check:

ssh wooo@192.168.0.110 '
for u in $(systemctl list-units "actions.runner.*" --all --no-legend --plain | awk "{print \$1}"); do
  echo "=== $u ==="
  systemctl show "$u" -p ActiveState -p SubState -p CPUQuotaPerSecUSec -p MemoryMax -p WatchdogUSec -p NRestarts
done
'

If WatchdogUSec is not 0, apply the guardrail script manually with sudo:

sudo /home/wooo/scripts/apply-runner-systemd-guardrails.sh --apply

6. P1 120/121 K3s

K3s must wait for 188 PostgreSQL and 110 Harbor.

6.1 Startup order

  1. 120 k3s.service
  2. 121 k3s.service, k3s-agent.service, or its live role
  3. CNI / kube-proxy
  4. Nodes Ready
  5. Core pods
  6. awoooi-prod pods
  7. keepalived VIP 192.168.0.125
  8. NodePorts 32334 and 32335

6.2 Checks

ssh wooo@192.168.0.120 '
hostname; uptime
pg_isready -h 192.168.0.188 -p 5432 || true
systemctl is-active k3s k3s-agent keepalived 2>/dev/null || true
kubectl get nodes -o wide 2>/dev/null || true
kubectl get pods -A 2>/dev/null | grep -v -E "Running|Completed" || true
kubectl get pods -n awoooi-prod -o wide 2>/dev/null || true
ip addr show | grep 192.168.0.125 || true
'

ssh wooo@192.168.0.121 '
hostname; uptime
systemctl is-active k3s k3s-agent keepalived 2>/dev/null || true
ip addr show | grep 192.168.0.125 || true
'

If K3s is activating while 188 PostgreSQL is down, fix PostgreSQL first. Restarting K3s repeatedly will not solve it.

6.3 120 / 121 AA / AS 與負載平衡判定

2026-06-12 15:19 live check 確認 120 / 121 都是 K3s control-plane，且兩台都是 k3s active、k3s-agent inactive。因此它們是 K3s 控制面 AA，不是傳統一主一從 AS。

但控制面 AA 不等於業務 workload AA。120 剛從 root filesystem fsck 恢復後，大多數 ArgoCD / AWOOOI / Velero / kube-system workload 仍集中在 121；120 主要只有 DaemonSet 類 Pod。每次 120 / 121 重啟或恢復後，都要額外跑 Pod 落點檢查：

ssh wooo@192.168.0.120 '
sudo kubectl get nodes -o wide
sudo kubectl get pods -A -o wide
sudo kubectl top nodes 2>/dev/null || true
sudo kubectl top pods -A --sort-by=cpu 2>/dev/null | head -30 || true
'

判定規則：

判定 條件 可宣稱
K3S_CONTROL_PLANE_AA 120 / 121 都是 Ready control-plane 控制面雙節點可用
WORKLOAD_IMBALANCED 主要 deployment / pod 都落在單一節點 不可宣稱服務 AA；需排程治理
WORKLOAD_BALANCED replicas >= 2 的核心 API / Web 跨 120 / 121 分散 可宣稱承載層分散
STATEFUL_AA storage replication、backup / restore drill、failover drill 都通過 才可宣稱資料層 AA

負載平衡與遷移評估的正式基準文件是 docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md。恢復期先完成 P0 備份鏈與 cold-start scorecard，再做 topology spread 或服務搬遷。

7. P2 AWOOOI Workloads

Run after K3s nodes are Ready:

ssh wooo@192.168.0.120 '
kubectl get deploy -n awoooi-prod
kubectl get pods -n awoooi-prod -o wide
kubectl get svc -n awoooi-prod
kubectl get events -n awoooi-prod --sort-by=.lastTimestamp | tail -40
'

curl -s --max-time 8 http://192.168.0.125:32334/api/v1/health
curl -s -o /dev/null -w "web=%{http_code}\n" --max-time 8 http://192.168.0.125:32335/

If pods are ImagePullBackOff, go back to 110 Harbor.

If API health fails because DB/Redis is down, go back to 188.

8. P2 Alert Chain

Current main path:

Prometheus/Alertmanager on 110
  -> http://192.168.0.125:32334/api/v1/webhooks/alertmanager
  -> AWOOOI API
  -> TelegramGateway
  -> Telegram

Alertmanager health alone is not enough. Run E2E:

curl -s -X POST http://192.168.0.125:32334/api/v1/webhooks/alertmanager \
  -H 'Content-Type: application/json' \
  -d '{"receiver":"cold-start-test","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"ColdStartE2ETest","severity":"info"},"annotations":{"summary":"Cold start E2E test, ignore"},"startsAt":"2026-05-05T11:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":""}],"groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"cold-start-test"}'

Expected: API returns success and Telegram receives the test alert.

9. P2 Schedules And Delayed Work

Do not mark the reboot complete until scheduled work is proven runnable. A container can be healthy while its cron path is broken.

Host / Layer Required check Success baseline
188 cron systemctl is-active cron and crontab -l cron active; backup, restart exporter, stats exporter entries present
188 backup-from-110 backup_110_last_success_timestamp in textfile/Prometheus last success age < 25h
188 momo-scheduler docker inspect momo-scheduler and docker logs --since 6h momo-scheduler container running healthy; 全部排程任務已註冊; Google Drive auth works; dashboard URLs use container-reachable hostnames
188 momo import manual run_auto_import_task() after parser changes selected sheet is 即時業績明細; imported date range has matching rows in daily_sales_snapshot and realtime_sales_monthly
110 cron systemctl is-active cron cron active; Docker/systemd textfile exporters fresh
110 startup units systemctl --failed zero failed units; stale momo-startup-complete and wooo-staggered-startup disabled
120 K8s CronJobs kubectl get cronjobs -n awoooi-prod unsuspended; no failed Jobs remain after current validation
121 DR drill crontab -l DR drill cron present unless explicitly paused

Useful checks:

ssh ollama@192.168.0.188 'systemctl is-active cron; crontab -l; ls -l /home/ollama/node_exporter_textfiles/*.prom'
ssh wooo@192.168.0.110 'systemctl --failed --no-pager; systemctl is-active cron; crontab -l'
ssh wooo@192.168.0.120 'sudo kubectl get cronjobs,jobs -n awoooi-prod'
ssh wooo@192.168.0.121 'systemctl is-active cron; crontab -l'

If a schedule succeeds but emits a false verification alert, fix the verification rule before releasing AI auto-remediation. False positives train operators to ignore real alarms.

10. P2/P3 Stateful Service Guardrails

Tier Examples Automation
BLOCK PostgreSQL data dir, ClickHouse data dir, Harbor DB, Sentry DB No automatic destructive action. Human approval only.
CRITICAL_HITL Redis, Kafka, MinIO, SignOz ClickHouse, Sentry ClickHouse Human-in-the-loop restart/repair.
STANDARD_HITL API/Web/worker, OpenClaw, litellm Restart only with evidence and blast-radius check.
AUTO Stateless exporters, blackbox, nginx exporter Auto restart allowed after verification.

Never use generic docker restart $(docker ps -q) during cold start.

10.1 Dirty-Reboot Storage Corruption

Treat these log signatures as storage corruption, not ordinary service flakiness:

  • Bad message
  • Structure needs cleaning
  • Unknown codec
  • PANIC: could not locate a valid checkpoint record
  • Kafka Malformed line in checkpoint files
  • ClickHouse broken and needs manual correction

Cold-start automation may stop a restart storm and collect evidence, but it must not delete the original data directory. If a filesystem returns Bad message or Structure needs cleaning, the real root cause is below the container layer. Online recovery can restore service from readable data, but complete historical recovery requires an offline filesystem check or backup restore.

10.2 ClickHouse Clean-Clone Recovery Pattern

Use this pattern for Sentry ClickHouse or SignOz ClickHouse when individual corrupted parts cannot be moved because the host filesystem rejects reads.

1. Stop the compose stack or at least stop dependent consumers.
2. Disable restart loops for the failing container.
3. Save logs and build an exclude list from unreadable store paths.
4. Preserve the original volume as _data.corrupt-YYYYMMDD-HHMMSS.
5. Create a clean _data clone with readable files only.
6. Add flags/force_restore_data.
7. Start ClickHouse first, then web/API, then consumers.
8. Verify HTTP, merge backlog, and restart count before releasing high-load services.

Do not replace this with rm -rf store/... unless the unreadable path is already backed up or the commander explicitly accepts data loss. The preferred incident artifact is:

/var/lib/docker/volumes/<volume>/_data.corrupt-YYYYMMDD-HHMMSS
/var/backups/<service>-<component>-YYYYMMDD-HHMMSS

10.3 Kafka Checkpoint Recovery Pattern

If Kafka refuses to start with malformed checkpoint files after a dirty reboot, preserve and move only checkpoint files:

log-start-offset-checkpoint
recovery-point-offset-checkpoint
replication-offset-checkpoint

Then start Kafka and confirm health before starting Snuba/Sentry consumers. Do not delete topic directories or Kafka logs during cold-start recovery.

11. P3 High-Load Services

Only release these after P0/P1/P2 gates are green:

Host Service Release condition
188 momo-scheduler / crawler load/core < 1.0 for 15 minutes and DB healthy
188 SignOz ClickHouse healthy and merge backlog trending down
188 litellm /health/liveliness good and provider route verified
110 Sentry Snuba consumers ClickHouse healthy and Kafka backlog decreasing
110 Sentry uptime-checker Sentry web/DB healthy
110 runners all previous gates green and load/core < 1.0 for 15 minutes

12. Baseline And AI Auto-Remediation Gate

12.1 Stable Runtime Baseline

These are release gates after the first cold-start recovery pass:

Area Baseline
188 host PostgreSQL accepting, Redis PONG, momo /health 200, SignOz HTTP reachable, load/core < 1.0 sustained before crawlers
110 host Harbor /v2/ 200/401, Gitea 200/302, Prometheus ready, Alertmanager healthy, Sentry HTTP 200/302/400, no ClickHouse/Kafka restart loop
K3s 120/121 nodes Ready, VIP 192.168.0.125 present, AWOOOI API 2xx/3xx, Web 2xx/3xx
Public routes https://awoooi.wooo.work/api/v1/health 2xx/3xx, https://mo.wooo.work/health 2xx/3xx
Guardrails Docker/systemd textfile exporters fresh, runner CPUQuota=200%, MemoryMax=2G, WatchdogUSec=0
Schedules cron active on 110/188/120/121; K8s CronJobs unsuspended; no current failed Jobs; 188 backup success < 25h
Backlog ClickHouse merges and Kafka/Snuba lag trending down, not increasing for two consecutive checks

If service health is green but load average remains high, check live CPU and IO before changing memory limits. High load after Sentry/Snuba or ClickHouse startup can be backlog drain; high CPU from runners/builds/crawlers is a release-order problem.

12.2 AI Auto-Remediation Gate

AI auto-repair can move from observe-only to limited execution only after:

  • Prometheus rules are loaded.
  • docker/systemd textfile exporter files are fresh.
  • blackbox probes have stable results.
  • cron/CronJob schedule checks are green.
  • AWOOOI API /api/v1/health passes.
  • Alertmanager E2E webhook passes.
  • Redis/KM/playbook health is available.
  • No active restart storm.
  • Host load/core remains below 1.0 for 15 minutes.

Until then:

  • diagnose only
  • notify only
  • require human approval for remediation
  • no DB/ClickHouse/Harbor/Sentry destructive action
  • no generic restart action against stateful services

13. One-Command Readiness Script

13.1 Single Pass

Run this when you want one read-only snapshot:

bash scripts/reboot-recovery/full-stack-cold-start-check.sh

The script is read-only. It does not restart services, delete data, change memory/CPU limits, or patch Kubernetes. It reports gates:

  • P0-NETWORK
  • P0-188-DATA
  • P0-110-REGISTRY-OBSERVABILITY
  • P1-K3S
  • P2-WORKLOAD-ALERTCHAIN
  • P2-PUBLIC-ROUTES
  • P2-SCHEDULES
  • runner guardrail state inside P0-110-REGISTRY-OBSERVABILITY

If it prints BLOCKED, fix the first blocked gate before moving forward.

13.2 Professional Watch Mode

Run this after a full reboot when you want the machine to keep checking until the whole stack is ready:

bash scripts/reboot-recovery/full-stack-cold-start-check.sh \
  --watch \
  --interval 60 \
  --max-attempts 30 \
  --send-alert-test

This is the standard next-reboot release command. It checks every 60 seconds for up to 30 attempts and exits only when the stack is GREEN or the last attempt remains degraded/blocked.

Use --send-alert-test for final release because the alert chain is not proven until the AWOOOI Alertmanager webhook accepts a real POST. Without --send-alert-test, the script intentionally leaves a warning so operators do not falsely mark alerting as complete.

13.3 Persistent Read-Only Monitor

After recovery, host 110 should run the same gate as a node-exporter textfile monitor:

bash scripts/reboot-recovery/install-cold-start-monitor-110.sh

This installs two scripts under /home/wooo/scripts/, adds a marked user-cron block, and writes:

  • /home/wooo/node_exporter_textfiles/cold_start_recovery.prom
  • /home/wooo/reboot-recovery/cold-start-last.log

The cron path uses --monitor-read-only, so it does not POST Alertmanager smoke events every 10 minutes. It converts the cold-start gate into Prometheus metrics:

  • awoooi_cold_start_monitor_up
  • awoooi_cold_start_pass_gates
  • awoooi_cold_start_warn_gates
  • awoooi_cold_start_blocked_gates
  • awoooi_cold_start_last_run_timestamp
  • awoooi_cold_start_last_green_timestamp
  • awoooi_cold_start_last_result{result="green|degraded|blocked|check_failed"}

Prometheus rules in ops/monitoring/alerts-unified.yml alert when the monitor is missing, stale, blocked, degraded, or has not been green for more than 6 hours.

13.4 Script-To-SOP Coverage Map

Script gate SOP coverage Blocks
P0-NETWORK host reachability, ARP, SSH every later phase
P0-188-DATA PostgreSQL, Redis, momo, SignOz K3s, AWOOOI API, momo public site
P0-110-REGISTRY-OBSERVABILITY Harbor, Gitea, Prometheus, Alertmanager, Sentry, runner quotas image pulls, CD, alert rules, runners
P1-K3S 120/121 K3s, VIP, node readiness, pod health workload and webhook health
P2-WORKLOAD-ALERTCHAIN AWOOOI API/Web, Alertmanager webhook AI auto-remediation and alert confidence
P2-PUBLIC-ROUTES external AWOOOI and momo URLs external release
P2-SCHEDULES cron, CronJobs, backups, textfile exporters, DR drill final done criteria

13.5 Next-Reboot Operator Contract

  1. Run the watch command above.
  2. If it stops at BLOCKED, repair the first blocked gate and rerun watch mode.
  3. If it stops at WARN, do not release runner/CD/AI full execution; clear or explicitly accept each warning.
  4. Release high-load services only after GREEN and load/core stays below 1.0 for 15 minutes.
  5. Record the final output summary and any manual repair in docs/LOGBOOK.md.

13.6 2026-05-29 補充：188 Public Gateway 與備份告警

aiops.wooo.work 的 188 public gateway 不可再指向單一 192.168.0.120:31234/31235。120 失聯時這會讓 public route 直接 502。正式 baseline 必須走 K3s VIP：

location /api/ {
    proxy_pass http://192.168.0.125:32334/api/;
}

location /api/v1/ws {
    proxy_pass http://192.168.0.125:32334/api/v1/ws;
}

location / {
    proxy_pass http://192.168.0.125:32335;
}

變更來源必須是 infra/ansible/roles/nginx/templates/188-all-sites.conf.j2，再用 infra/ansible/playbooks/nginx-sync.yml 收斂；禁止只改 188 live 檔而不回寫 Ansible baseline。

備份告警有兩層，缺一不可：

  • ops/monitoring/alerts-unified.yml 是 repo canonical。
  • 110 live /home/wooo/monitoring/alerts.yml 與 /home/wooo/monitoring/alerts-unified.canonical.yml 必須一致，否則 prometheus-rule-drift-guard 可能把規則拉回舊版。

重啟後必查：

curl -s http://127.0.0.1:9090/api/v1/rules \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); names=[r.get("name") for g in d["data"]["groups"] for r in g["rules"]]; print([n for n in ["BackupAggregateRunFailed","BackupConfigCapturePartial","BackupOffsiteCopyStale","BackupCredentialEscrowEvidenceMissing","ColdStartRecoveryBlocked"] if n not in names])'

cat /home/wooo/node_exporter_textfiles/prometheus_rule_drift_guard.prom

若 120 尚未恢復，BackupConfigCapturePartial{target="120-k3s-host-configs"} 與 cold-start blocked 是正確訊號，不可消音。120 恢復後再重跑：

/backup/scripts/backup-configs.sh
/backup/scripts/backup-all.sh
/backup/scripts/sync-offsite-backups.sh --mode sync
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color

13.7 2026-05-29 補充：momo PostgreSQL Index 與資料同步

mo.wooo.work 不能只看 /health 或首頁 200。重啟或 fsck 後，PostgreSQL index 可能讓匯入流程表面完成，但 daily_sales_snapshot 未同步到 realtime_sales_monthly。本次症狀：

  • daily_sales_snapshot 已有 2026-05-01 到 2026-05-28 的 17,353 筆。
  • realtime_sales_monthly 同日期範圍為 0 筆。
  • momo-scheduler log 出現 PostgreSQL 內部錯誤 posting list tuple ... cannot be split。

標準處理順序：

# 188 / momo-db，只重建索引，不刪資料
docker exec -i momo-db bash -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1' <<'SQL'
REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;
SQL

重建索引後，才可針對缺漏日期做 idempotent 補同步。正式作法必須先確認 realtime_sales_monthly 該日期範圍筆數，若非 0，需先保存查詢結果並確認是否重跑同範圍同步；不可整表 truncate、不可整庫 restore。補同步後至少驗證：

SELECT count(*), min(snapshot_date::date), max(snapshot_date::date)
FROM daily_sales_snapshot
WHERE snapshot_date::date BETWEEN DATE '2026-05-01' AND DATE '2026-05-28';

SELECT count(*), min("日期"::date), max("日期"::date)
FROM realtime_sales_monthly
WHERE "日期"::date BETWEEN DATE '2026-05-01' AND DATE '2026-05-28';

兩張表同日期範圍筆數與日期上下界必須一致。完成後清除 momo 應用 cache：

docker exec momo-pro-system python -c 'from services.cache_service import clear_all_cache; clear_all_cache(); print("cache_cleared")'

14. 主機開機、關機、重啟 SOP

本節是每次 110 / 120 / 121 / 188 相關電源操作的標準程序。112 是 Kali，只保留 read-only evidence，不納入本輪恢復或例行重啟釋出。

14.1 共同紅線

類型 禁止事項 正確處理
120 offline 不可消音 ColdStartHost120Unreachable、ColdStartRecoveryBlocked 或 120 config backup alert 保留紅燈，直到 console/VM recovery 後重跑完整 chain
Filesystem 不可對已掛載 root filesystem 做 online fsck 只在 console/rescue/initramfs 狀態下離線修復
Backup 不可用單項 backup 成功宣稱 aggregate backup green 以 backup-all、offsite verifier、cold-start scorecard 三者共同判定
Credential 不可把密碼、token、private key 寫進 repo、LOGBOOK 或聊天 只寫 non-secret evidence marker / vault reference
Stateful data 不可 truncate、DROP、整庫 restore 或整批刪 volume 先保存證據，優先 REINDEX TABLE CONCURRENTLY / clean-clone / idempotent resync
Automation 不可在 P0/P1 未綠時釋出 runner/CD/AI full execution observe-only，runner/CD 最後釋出

14.2 關機前 SOP

目標是保留證據、停止高負載來源、讓 stateful service 乾淨落地。

  1. 宣告維護窗口，建立 docs/LOGBOOK.md 重啟紀錄草稿。
  2. 跑 preflight snapshot：
/home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1
/backup/scripts/backup-status.sh --no-notify
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
  1. 保存 host reboot evidence：
for h in 110 120 121; do
  ssh wooo@192.168.0.$h 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20; systemctl --failed --no-pager' || true
done
ssh ollama@192.168.0.188 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20; systemctl --failed --no-pager' || true
  1. 暫停高負載與自動化釋出：
順序 對象 操作原則
1 Gitea / actions runners 停止新 job；不要在 build 中途硬關，先讓可完成 job 結束或人工取消
2 AI auto-remediation 切 observe-only；禁止自動 restart stateful services
3 momo crawler / scheduler / heavy batch 暫停會啟動 Chrome、批次匯入或大量 DB 寫入的工作
4 Sentry/Snuba/ClickHouse heavy consumers 確認沒有 restart storm；必要時 controlled stop
5 K3s workload 優先 drain / cordon 可達節點；不可在 120 已不可達時假裝 drain 完成
  1. 全機關機順序：
1. runner/CD and high-load batch
2. AI auto-remediation execution
3. AWOOOI workload layer
4. 121 K3s agent side
5. 120 K3s server side
6. 110 registry / observability, after evidence and backup status are captured
7. 188 data layer last
8. network / UPS / hypervisor last, if applicable

188 必須最後關，因為 PostgreSQL / Redis / momo DB / K3s datastore 是其他層的共同依賴。

14.3 開機 SOP

開機順序固定走 dependency chain，不追最吵的 alert。

1. Physical network: switch, NIC, ARP, SSH
2. 188 data layer: PostgreSQL, Redis, Docker, momo DB, SignOz dependencies
3. 110 registry / observability: Harbor, Gitea, Prometheus, Alertmanager, Sentry
4. 120 K3s server / VIP path
5. 121 K3s agent / failover path
6. AWOOOI API/Web workload
7. Public routes and Alertmanager E2E
8. Backups, cron, CronJobs, textfile exporters
9. momo scheduler / crawlers and high-load consumers
10. runners/CD
11. AI auto-remediation limited execution

開機後每一層都要有 live evidence。最小驗收命令：

for h in 110 120 121 188; do
  ping -c 2 -W 2 192.168.0.$h >/dev/null && echo "PING_OK 192.168.0.$h" || echo "PING_FAIL 192.168.0.$h"
  nc -G 3 -z 192.168.0.$h 22 && echo "SSH_OK 192.168.0.$h" || echo "SSH_FAIL 192.168.0.$h"
done

ssh ollama@192.168.0.188 'systemctl is-active docker postgresql@14-main redis-server nginx || true; pg_isready -h localhost -p 5432 || true; docker ps --format "{{.Names}}\t{{.Status}}" | head -80'
ssh wooo@192.168.0.110 'systemctl is-active docker cron || true; curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true; curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true'
ssh wooo@192.168.0.121 'sudo kubectl get nodes -o wide; sudo kubectl get pods -A | grep -v -E "Running|Completed" || true'

/backup/scripts/backup-status.sh --no-notify
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
/home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1

14.4 單主機重啟 SOP

Host 重啟前條件 重啟後必查 完成條件
110 不在 backup-all / rclone / verify window；runner job 已停止或人工取消；188 healthy Docker, Harbor, Gitea, Prometheus, Alertmanager, Sentry, cron, textfile exporters, /backup/scripts/backup-status.sh --no-notify 110 services green；backup status 沒有新增 stale / failed；runner/CD 最後釋出
120 必須是 console-first 維護；若可達，先 cordon/drain；若不可達，不宣稱 drain 成功 power/VM/NIC/boot/initramfs/fsck state, SSH, kubectl get nodes, SchedulingDisabled 清除狀態 120 ping/SSH OK；mon Ready；backup configs/all/offsite/verify/cold-start chain 重跑
121 120 / 188 healthy；可達時先 cordon/drain k3s-agent 或 live role、VIP 狀態、kubectl get nodes, pod placement mon1 Ready；VIP / NodePort 路徑正常；workload 無新增 failed pods
188 110 backup status 已保存；停止或延後 momo heavy import；確認無 DB restore / migration PostgreSQL, Redis, Docker, momo DB parity, SignOz/ClickHouse, cron, backup freshness DB accepting；momo parity 綠；188 backup jobs fresh；高負載服務最後釋出

14.4.1 110 重啟後恢復指揮卡

110 是 registry / observability / backup center。重啟後先看 host 與核心端口，不要第一時間重啟 Docker daemon。

順序 檢查 成功基準 失敗處理
1 systemctl is-system-running / failed units / Swap running、failed 0 或可解釋、Swap 未持續增加 先分辨 stale unit、現役 service、storage/network 問題
2 Docker daemon systemctl is-active docker=active 若 Docker activating，先看 journal；不要連續 restart/kill
3 Harbor / registry local /v2/ 回 200/401，public registry 未登入 401 只針對失效 upstream 做最小修復；避免 daemon restart
4 Gitea / runners Gitea 200/302；runner 最後釋出 runner job 不可在 P0/P1 未綠時搶資源
5 Prometheus / Alertmanager /-/ready、/-/healthy OK；required alerts visible 若告警缺失，先修 rules/drift guard，再談自動化
6 Sentry / Langfuse / Stock / public tools public 2xx/3xx；container 非 restart loop 只修明確故障服務；不要 compose 全套重建
7 backup / offsite backup-status --no-notify、offsite verifier 120 不可達時 Configs 紅燈保留

110 post-reboot 最小命令：

ssh wooo@192.168.0.110 '
date; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true
free -h; swapon --show
systemctl is-active docker cron || true
curl -s -o /dev/null -w "harbor_v2=%{http_code}\n" --max-time 5 http://127.0.0.1:5000/v2/ || true
curl -s -o /dev/null -w "gitea=%{http_code}\n" --max-time 5 http://127.0.0.1:3001/ || true
curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true
curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true
docker ps --format "{{.Names}}\t{{.Status}}" | head -120
'

2026-06-12 補充：stockplatform-shared-ui-monitor.timer 指向不存在的 legacy path 時，可停用 stale timer 解除 host failed unit；但正式 source-of-truth 必須後續清理，不能把 reset-failed 當修復。

14.4.2 188 重啟後恢復指揮卡

188 是資料與 AI/Web 依賴主機。它恢復前，不釋出 K3s、AWOOOI API、momo heavy import 或 AI auto-remediation。

順序 檢查 成功基準
1 PostgreSQL pg_isready accepting，無 checkpoint / WAL panic
2 Redis PONG
3 Docker / containerd active；momo-db / signoz / openclaw / litellm 非 restart loop
4 momo DB parity daily_sales_snapshot 與 realtime_sales_monthly 目前月份筆數與日期上下界一致
5 SignOz / monitoring bridge HTTP 200；ClickHouse 不在修復風暴
6 momo scheduler container healthy，recent activity pattern > 0；heavy import 等 DB green 後釋出
7 backup freshness 188 backup textfile / 110 backup-from-188 freshness OK

188 post-reboot 不可用「首頁 200」取代 DB parity；若出現 posting list tuple ... cannot be split，只走 REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;，不可 truncate 或整庫 restore。

14.4.3 120 恢復指揮卡

120 目前是 console-first blocker。它不可達時，遠端只能做證據收集，不能假裝修復。

狀態 判定 正確動作
ping / SSH / ARP 全失敗 host / VM / network 層未知 到 hypervisor / console 確認 power、NIC、boot screen
initramfs / fsck prompt filesystem repair gate 依 120-fsck-maintenance-checklist.sh 離線處理
SSH 恢復但 K3s NotReady K3s / runtime 層 先看 journalctl -u k3s、containerd、188 PostgreSQL，再解除 cordon
node Ready 但 SchedulingDisabled 調度狀態未解除 確認健康後 kubectl uncordon mon，再看 workload

120 恢復後不得只看 kubectl get nodes。必須強制補跑：

/backup/scripts/backup-configs.sh
/backup/scripts/backup-all.sh
/backup/scripts/sync-offsite-backups.sh --mode sync
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
/home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1

14.4.4 121 重啟後恢復指揮卡

121 是 K3s failover / secondary control-plane path。它重啟後的核心是「不要讓 mon1 Ready 掩蓋 mon 不可達」。

檢查 成功基準 注意
SSH / systemd host ready、failed units 可解釋 121 green 不代表 120 green
K3s role kubectl get nodes -o wide 可讀 若只剩 mon1 Ready，仍是 degraded cluster
VIP / NodePort VIP / public routes 通 必須確認 route 走 192.168.0.125:32334/32335
Cron / DR drill cron present、DR drill 未被誤停 schedule green 是 cold-start done criteria 的一部分

若 121 重啟後看到 mon1 Ready 但 mon NotReady,SchedulingDisabled，結論是「121 恢復，cluster 仍 degraded」，不可把 121 正常誤報成 K3s 全綠。

14.5 每次重啟紀錄格式

每次開機、關機、重啟都要在 docs/LOGBOOK.md 追加紀錄，並把必要狀態同步到本 SOP 或 workplan。

## YYYY-MM-DD | Host reboot / shutdown / startup record

Scope:
- Hosts:
- Operation: shutdown / startup / reboot / recovery
- SOP version used:
- Operator:
- Maintenance window:

Pre-check:
- Cold-start scorecard:
- Backup status:
- Offsite verifier:
- Public routes:
- momo DB parity:
- Alertmanager rules / E2E:
- Credential escrow:

Execution:
- Start time:
- End time:
- Commands / console actions:
- Services paused:
- Services released:

Result:
- 110:
- 120:
- 121:
- 188:
- Cold-start scorecard after:
- Backup status after:
- Offsite verifier after:
- DB parity after:
- Alerts after:

Difference versus previous reboot:
- Faster:
- Slower:
- New blocker:
- Repeated blocker:
- False positive / detector tuning:
- SOP change required: yes/no

SOP update:
- Previous version:
- New version:
- Change reason:
- Files updated:

14.6 SOP 版本比較與改版規則

每次重啟後必須比較上一次紀錄，不只寫「已恢復」。

比較項 判定方式
Time to SSH 從 power-on 到各 host SSH OK
Time to K3s Ready 從 120/121 boot 到 nodes Ready
Time to public routes 從 K3s Ready 到 public 2xx/3xx
Time to backup green 從 110 ready 到 backup status / offsite verifier green
Persistent blockers 連續兩次以上出現即列入 SOP hard gate
False positives 例如 momo scheduler detector WARN；要寫清楚直接證據與調整方向
Procedure drift live cron、Ansible template、script path 與 SOP 不一致時，先修 canonical，再修 SOP

改版規則：

  • 只更新 live baseline 或百分比：不升版，只更新日期與 evidence。
  • 新增、刪除或改變操作順序：升 minor version，例如 v1.4 -> v1.5。
  • 牽涉破壞性操作、資料修復策略或人為批准邊界：升 major-ready review，先經人工批准。

14.7 2026-06-06 重啟紀錄比較錨點

2026-06-06 沒有執行新重啟；本次是 live recovery check。它仍要作為下一次重啟比較基準：

項目 2026-06-06 baseline
Overall 65% BLOCKED
Cold-start PASS=71 WARN=3 BLOCKED=3
Remaining hard blocker 120 ping / SSH / K3s read-only check
Backup aggregate failed=1, Configs only, due 120 config capture
Backup freshness 110 and 188 fresh, no stale jobs
Offsite 13 repos latest-only green
Escrow 5 markers missing
momo scheduler direct healthy; 15:03 scorecard no longer emits scheduler WARN

14.8 2026-06-12 重啟後比較錨點

2026-06-12 110 被非計畫重啟後，本 SOP v1.5 的新比較錨點如下：

項目 2026-06-12 post-reboot baseline
110 host systemd running，failed units 0，Swap 0B/7.8GiB
110 service recovery Harbor / Gitea / Prometheus / Alertmanager / Sentry / Stock / public tools reachable
Cold-start PASS=72 WARN=2 BLOCKED=3
Remaining hard blocker 120 ping / SSH / K3s read-only check
WARN 120-driven backup aggregate/config component and 120 K3s schedule check
Backup freshness 110 13/13 fresh failed=1，188 2/2 fresh failed=0，stale none
Offsite 13 repos latest-only green，REMOTE_LATEST_ONLY_OK=1，VERIFY_OK=1
Alerts Prometheus and Alertmanager expose all five required backup/cold-start/escrow alerts
momo scheduler scorecard reads SCHEDULER_RECENT_ACTIVITY 1070 after detector fix
SOP change v1.5 adds startup judgment layers, GO/NO-GO tree, host recovery cards, and timeline checks

14.9 2026-06-13 CD 後恢復比較錨點

2026-06-13 不是主機重啟，而是用來驗證「120/121 workload balancing + CD known_hosts guardrail」是否能承受下一次正常部署的比較錨點。

項目 2026-06-13 14:16 baseline
Gitea / ArgoCD Gitea main a520c32d，deploy marker a520c32d，ArgoCD revision a520c32d，sync Synced
K3s image readback API/Web/Worker image tag e897c8bf20125f7792f9aee21d7f503f4358bcfb
K3s placement API/Web verified split across mon / mon1 after the latest deploy marker；Worker single replica healthy
Cold-start PASS=83 WARN=0 BLOCKED=0
Public routes Scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS
Backup backup-status: 110 13/13 fresh failed=0，188 2/2 fresh failed=0，core_blockers=0，escrow_missing=5
Offsite textfile remote_verify_ok=1、full_verify_fresh=1，13 repos each snapshot_count=1
SSH trust Global known_hosts retained 120 / 188 entries after CD; deploy-specific trust moved to deploy_known_hosts
Remaining non-service debt km-vectorize waits for official 03:00 lastSuccessfulTime; credential escrow missing count remains 5; 188 host degraded only by certbot units
SOP change v1.9 keeps CD / SSH trust guardrail, workload-balance proof, DR escrow red gate, and km-vectorize official schedule gate in the first-screen declaration rules

14.10 重啟後時間軸驗證

每次重啟後照時間軸推進，不要等到最後才一次判定。

時間點 目標 必跑證據 可以宣稱
T+0 power / VM / console 已開始 console / hypervisor / UPS / operator note maintenance started
T+5m LAN / SSH 回復 ping、ARP、SSH port、who -b HOST_BOOTED
T+15m 主機基礎服務回復 systemctl is-system-running、failed units、Docker / PostgreSQL / Redis / K3s role checks HOST_READY
T+30m 核心服務回復 188 DB、110 Harbor/Gitea/Prom/AM、K3s nodes、AWOOOI API/Web、public routes SERVICE_READY for scoped hosts
T+45m 排程與資料一致性 backup status、offsite verifier、momo DB parity、CronJobs、alert visibility service recovery confidence
T+60m 釋出高負載與自動化 cold-start scorecard、load/core、runner guardrails、AI observe-only gate release runner/CD only if gates allow

若任一時間點卡住，記錄卡在哪個 gate，不要跳到下一層。連續兩次重啟都卡同一 gate，必須回寫 §16 Known Drift 或 workplan。

15. Done Criteria

All must be true:

  • Four hosts reachable by SSH.
  • 188 PostgreSQL and Redis healthy.
  • 110 Harbor, Gitea, Prometheus, Alertmanager healthy.
  • 120/121 K3s nodes Ready.
  • VIP 192.168.0.125 present.
  • AWOOOI API and Web reachable through NodePort/VIP.
  • Alertmanager E2E webhook succeeds.
  • cron/CronJob schedules are active, unsuspended, and verified.
  • momo daily_sales_snapshot 與 realtime_sales_monthly 在最新匯入日期範圍內筆數一致。
  • Sentry and SignOz are either healthy or explicitly in controlled backlog recovery.
  • High-load batch services are capped or delayed.
  • Runners are guarded and released last.
  • AI auto-remediation is not in full execution mode until all gates are green.
  • 110 cold-start textfile monitor is fresh, and Prometheus has the cold-start alert rules loaded.
  • 110 global /home/wooo/.ssh/known_hosts still contains verified 120 / 188 entries after any CD run; deploy jobs use /home/wooo/.ssh/deploy_known_hosts only.

15.1 可宣稱狀態

可宣稱文字 必要條件
110 host recovered 110 HOST_READY，failed units 0 或全部可解釋，核心端口與 cron / backup status 已查
public core services recovered public routes/TLS 2xx/3xx，AWOOOI API health、Harbor/Gitea/Stock/Sentry/SignOz/Langfuse/Bitan smoke OK
backup/offsite current backup-status --no-notify 無 stale，offsite verifier VERIFY_OK=1，且任何 failed component 有明確 owner
service recovery with known blocker cold-start BLOCKED 只剩已知 blocker，例如 120；告警保持可見
full-stack green §15 全部成立，cold-start WARN=0 BLOCKED=0
DR complete full-stack green 且 credential escrow missing count 為 0

16. Known Drift To Fix After Recovery

這些項目必須在事故後整理，不要在 P0 恢復中途順手大改。

  • SERVICE-ENDPOINTS.md still has old Prometheus/Alertmanager locations.
  • Audit older docs for direct node webhook targets; current main path should be VIP 192.168.0.125:32334.
  • OpenClaw 8088 vs 8089 must be live-confirmed and normalized.
  • 188 compose paths drift between /home/ollama/* and Ansible /opt/*.
  • 110 runner docs still mention Docker runner in places; live startup prefers host gitea-act-runner-host.service.
  • scripts/setup-runner-watchdog.sh conflicts with the 2026-05-05 runner watchdog disablement guardrail.
  • grist.wooo.work / registry.wooo.work public HTTP/HTTPS currently route to aiops.wooo.work; their old 110 certbot renewal configs are disabled until public routing is corrected or DNS-01 renewal is configured.
  • stockplatform-shared-ui-monitor.timer / service source-of-truth 仍需清理或重建；2026-06-12 只停用 stale timer 以解除 host degraded。
  • 111 local Ollama fallback 目前不可達；production provider 由 GCP-A / GCP-B 承接，但 111 恢復應另列 AI provider resilience 工作。
  • 本 SOP v1.5 新增內容已用繁體中文補強；舊章節仍有英文段落，後續 runbook hygiene 應分批翻譯，不要在事故 P0 中混入大規模格式重排。
Reference in New Issue View Git Blame Copy Permalink