docs(ops): codify full stack cold start recovery

docs/LOGBOOK.md

@@ -3269,3 +3269,48 @@ DATABASE_URL=postgresql+asyncpg://u:p@localhost:5432/test REDIS_URL=redis://loca
   apps/api/tests/test_openclaw_alert_cloud_fallback_gate.py -q
 # 15 passed
 ```
+
+---
+
+## 2026-05-06（台北）— 全棧重開機冷啟動 SOP / baseline / watch mode
+
+**觸發**：2026-05-05 晚間 110 / 120 / 121 / 188 異常重開機後，要求把本次恢復順序、服務相依、放行邏輯、最後確認機制完整文件化，並建立下次重開機可快速恢復的標準做法。
+
+### 已完成
+
+| Artifact | Result |
+|----------|--------|
+| `docs/runbooks/FULL-STACK-COLD-START-SOP.md` | 升級為 v1.1，補齊 Golden Startup Order、Mermaid 依賴圖、phase gate 邏輯、script-to-SOP 覆蓋表、next-reboot operator contract |
+| `ops/reboot-recovery/full-stack-cold-start-baseline.yml` | 新增機器可讀 baseline，固定 hosts、roles、啟動順序、endpoint code、schedule freshness、stateful-service 禁區、AI auto-remediation gate |
+| `scripts/reboot-recovery/full-stack-cold-start-check.sh` | 新增 `--watch` / `--interval` / `--max-attempts`，可在重開機後反覆檢查直到 `GREEN` |
+
+### 標準下次重開機放行指令
+
+```bash
+bash scripts/reboot-recovery/full-stack-cold-start-check.sh \
+  --watch \
+  --interval 60 \
+  --max-attempts 30 \
+  --send-alert-test
+```
+
+### 驗證結果
+
+```bash
+bash -n scripts/reboot-recovery/full-stack-cold-start-check.sh
+# OK
+
+ruby -e 'require "yaml"; YAML.load_file("ops/reboot-recovery/full-stack-cold-start-baseline.yml"); puts "YAML OK"'
+# YAML OK
+
+bash scripts/reboot-recovery/full-stack-cold-start-check.sh --watch --interval 1 --max-attempts 1 --send-alert-test
+# PASS=50 WARN=0 BLOCKED=0
+# Result: GREEN. Full stack is ready for controlled runner/CD release.
+```
+
+### 放行原則
+
+- `BLOCKED`：停止釋放後續 phase，先修第一個阻塞 gate。
+- `WARN`：不可釋放 runner/CD/AI full execution，需清掉或明確接受警告。
+- `GREEN`：只代表可進入下一階段；高負載 crawler / Snuba / ClickHouse merge / runner/CD 仍需最後釋放。
+- Stateful DB / ClickHouse / Kafka / Harbor / Sentry 資料層不可由 AI 自動破壞性修復。

docs/runbooks/FULL-STACK-COLD-START-SOP.md

@@ -1,7 +1,7 @@
 # AWOOOI Full-Stack Cold Start SOP
 
-> Version: v1.0
-> Last updated: 2026-05-05 Asia/Taipei
+> Version: v1.1
+> Last updated: 2026-05-06 Asia/Taipei
 > Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
 
 ---
@@ -38,6 +38,71 @@ The rule is simple: **recover the dependency chain, not the loudest symptom.**
 
 Never start runner/CD before 188 PostgreSQL, 110 Harbor, K3s nodes, and AWOOOI API are healthy.
 
+### 1.1 Dependency Graph
+
+```mermaid
+flowchart TD
+  network["P0 network: LAN, ARP, SSH"] --> data188["188 data: PostgreSQL, Redis, momo DB, SignOz"]
+  network --> obs110["110 registry/observability: Harbor, Gitea, Prometheus, Alertmanager, Sentry"]
+  data188 --> k3s["120/121 K3s: server, agent, VIP, NodePorts"]
+  obs110 --> k3s
+  k3s --> workload["AWOOOI workload: API, Web, K8s Secrets"]
+  workload --> alertchain["Alert chain: Alertmanager webhook, Telegram"]
+  workload --> public["Public routes: awoooi.wooo.work, mo.wooo.work"]
+  public --> schedules["Schedules: cron, CronJobs, backups, exporters"]
+  schedules --> highload["High-load release: crawlers, Snuba, ClickHouse merges, runners/CD"]
+  highload --> ai["AI auto-remediation: limited execution"]
+```
+
+This is also captured in the machine-readable baseline:
+
+```text
+ops/reboot-recovery/full-stack-cold-start-baseline.yml
+```
+
+The YAML baseline is the source of truth for:
+
+- hosts, roles, and SSH users
+- phase ordering
+- service startup dependencies
+- endpoint success codes
+- schedule freshness thresholds
+- stateful-service protection boundaries
+- AI automation release gates
+
+### 1.2 Phase Gate Logic
+
+Each phase has the same decision rule:
+
+| Result | Meaning | Action |
+|--------|---------|--------|
+| `BLOCKED` | A dependency required by later phases is down. | Stop phase release and fix the first blocked gate. |
+| `WARN` | Core dependency passed, but confidence is incomplete. | Continue diagnosis, but do not release runner/CD/AI full execution. |
+| `GREEN` | All checks in scope passed. | Release the next phase only. |
+
+The cold-start flow is intentionally conservative:
+
+```text
+P0 network green
+  -> P0 188 data green
+  -> P0 110 registry/observability green
+  -> P1 K3s green
+  -> P2 workload + alert chain green
+  -> P2 public routes green
+  -> P2 schedules green
+  -> P3 high-load services and runners/CD
+  -> AI auto-remediation limited execution
+```
+
+The final release condition is not "containers are running". It is:
+
+```text
+PASS > 0
+WARN = 0
+BLOCKED = 0
+Result: GREEN
+```
+
 ---
 
 ## 2. Automation Freeze
@@ -443,26 +508,63 @@ Until then:
 
 ## 13. One-Command Readiness Script
 
-Run:
+### 13.1 Single Pass
+
+Run this when you want one read-only snapshot:
 
 ```bash
 bash scripts/reboot-recovery/full-stack-cold-start-check.sh
 ```
 
-The script is read-only. It reports gates:
+The script is read-only. It does not restart services, delete data, change memory/CPU limits, or patch Kubernetes. It reports gates:
 
 - `P0-NETWORK`
 - `P0-188-DATA`
-- `P0-110-REGISTRY`
+- `P0-110-REGISTRY-OBSERVABILITY`
 - `P1-K3S`
-- `P2-WORKLOAD`
-- `P2-ALERTCHAIN`
+- `P2-WORKLOAD-ALERTCHAIN`
 - `P2-PUBLIC-ROUTES`
 - `P2-SCHEDULES`
 - runner guardrail state inside `P0-110-REGISTRY-OBSERVABILITY`
 
 If it prints `BLOCKED`, fix the first blocked gate before moving forward.
 
+### 13.2 Professional Watch Mode
+
+Run this after a full reboot when you want the machine to keep checking until the whole stack is ready:
+
+```bash
+bash scripts/reboot-recovery/full-stack-cold-start-check.sh \
+  --watch \
+  --interval 60 \
+  --max-attempts 30 \
+  --send-alert-test
+```
+
+This is the standard next-reboot release command. It checks every 60 seconds for up to 30 attempts and exits only when the stack is `GREEN` or the last attempt remains degraded/blocked.
+
+Use `--send-alert-test` for final release because the alert chain is not proven until the AWOOOI Alertmanager webhook accepts a real POST. Without `--send-alert-test`, the script intentionally leaves a warning so operators do not falsely mark alerting as complete.
+
+### 13.3 Script-To-SOP Coverage Map
+
+| Script gate | SOP coverage | Blocks |
+|-------------|--------------|--------|
+| `P0-NETWORK` | host reachability, ARP, SSH | every later phase |
+| `P0-188-DATA` | PostgreSQL, Redis, momo, SignOz | K3s, AWOOOI API, momo public site |
+| `P0-110-REGISTRY-OBSERVABILITY` | Harbor, Gitea, Prometheus, Alertmanager, Sentry, runner quotas | image pulls, CD, alert rules, runners |
+| `P1-K3S` | 120/121 K3s, VIP, node readiness, pod health | workload and webhook health |
+| `P2-WORKLOAD-ALERTCHAIN` | AWOOOI API/Web, Alertmanager webhook | AI auto-remediation and alert confidence |
+| `P2-PUBLIC-ROUTES` | external AWOOOI and momo URLs | external release |
+| `P2-SCHEDULES` | cron, CronJobs, backups, textfile exporters, DR drill | final done criteria |
+
+### 13.4 Next-Reboot Operator Contract
+
+1. Run the watch command above.
+2. If it stops at `BLOCKED`, repair the first blocked gate and rerun watch mode.
+3. If it stops at `WARN`, do not release runner/CD/AI full execution; clear or explicitly accept each warning.
+4. Release high-load services only after `GREEN` and load/core stays below `1.0` for 15 minutes.
+5. Record the final output summary and any manual repair in `docs/LOGBOOK.md`.
+
 ---
 
 ## 14. Done Criteria