fix(ops): harden cold-start schedule recovery

2026-05-05 22:14:54 +08:00
parent 10cd9fc025
commit 894174da5b
13 changed files with 1073 additions and 43 deletions
--- a/docs/LOGBOOK.md
+++ b/docs/LOGBOOK.md
@@ -6,6 +6,38 @@

 ---

+## 2026-05-05 | 重開機後排程與 startup baseline 修復
+
+**背景**：四台主機非預期重開機後，統帥要求確認所有服務、網站、工具、資料庫與排程都能正常恢復，不能只看容器 `healthy`。
+
+**本次排程/啟動鏈修補**：
+- 120/121 K3s 回到 Ready；CD workflow 目標從 121 改為 120，避免 121 worker kubeconfig `127.0.0.1:6443` 造成 Secrets patch 失敗；120 已驗證 limited sudo kubectl 可用。
+- K8s CronJob 修正：`k3s-status-report`、`weekly-report`、`km-vectorize` 改用存在的 service account、live API image、cluster service DNS；手動 job 驗證 drift/k3s/weekly 可完成，歷史 failed jobs 已清掉。
+- KM embedding schema 從 768/錯誤 typmod 修為 `vector(1024)`；原 embedding 已備份到 `knowledge_entries_embedding_backup_20260505`，正在以 `bge-m3:latest` 重建。
+- 188 momo backup script 修正 quote/validation/Telegram optional/error cleanup；成功產出 `/home/ollama/momo_backups/momo_analytics_20260505_212032.sql.gz`。
+- 188 `backup-from-110.sh` 因 SSH config 權限錯誤導致 `HostBackupFailed`；修正 `.ssh/config` 權限與 110 identity 設定後，以低優先權手動備份成功，Prometheus `backup_110_last_success_timestamp` 已更新。
+- 188 momo-scheduler 修正 dashboard URL：容器內改打 `http://momo-pro-system`，不再打 `127.0.0.1:5000`。
+- 188 Google Drive token 從 legacy pickle 轉為 JSON，scheduler 容器內 `GoogleDriveService().authenticate()` 通過。
+- 188 daily sales import 修正 Excel sheet 選擇，優先讀 `即時業績明細`；手動匯入成功 `19934` 筆，日期 `2026-04-01 ~ 2026-05-03`。
+- 188 import 尾端驗證修正：改比對本次匯入日期範圍，不再用全表筆數硬比；`daily_sales_snapshot` 與 `realtime_sales_monthly` 在該日期範圍皆 `19934` 筆且驗證通過。
+- 110 startup 修復：移除 `/etc/sysctl.conf` 中誤寫的非法敏感純文字行；`systemd-sysctl` 恢復成功。
+- 110 停用兩個過期 startup units：`momo-startup-complete.service`（指向不存在路徑/錯 host）與 `wooo-staggered-startup.service`（舊 GitLab 延遲啟動且會增加重開機負載）。
+- 110 `awoooi-startup-110.service` timeout 從 5 分鐘延長到 15 分鐘，重跑後 `ActiveState=active`、`SubState=exited`、`Result=success`，`systemctl --failed` 為 0。
+- 110 certbot timer 失敗追查：`grist.wooo.work` / `registry.wooo.work` public route 目前被導向 `aiops.wooo.work`，HTTP-01 無法從 110 成功；已將兩個 stale renewal config 移至 `/etc/letsencrypt/renewal-disabled-codex-*`，並 reset certbot failed state。憑證 archive 未刪除；後續需修 public route 或改 DNS-01。
+- `scripts/reboot-recovery/full-stack-cold-start-check.sh` 新增 `P2-SCHEDULES`，覆蓋 188/110/120/121 cron、textfile mtime、188 backup freshness、110 failed units、K8s CronJob/Job/Pod 狀態、121 DR drill cron。
+- `docs/runbooks/FULL-STACK-COLD-START-SOP.md` 新增排程驗證章節與 done criteria，要求排程真正可執行才算 reboot recovery 完成。
+
+**最終驗證**：
+- KM reembed 完成：`1774/1774` success、`0` failed；DB 目前 `knowledge_entries` total `1785`、embedded `1776`、vector dims `1024..1024`，舊 embedding backup `1691` rows。
+- 手動 `km-vectorize` CronJob `km-vectorize-codex-220715` 完成，回 `embed-all: 200 {"total":0,"success":0,"failed":0}`。
+- `bash scripts/reboot-recovery/full-stack-cold-start-check.sh --send-alert-test` → `PASS=50 WARN=0 BLOCKED=0`，包含 Alertmanager webhook E2E、public routes、cron/CronJob/textfile/systemd schedule checks。
+- Prometheus firing alerts 已從 `HostBackupFailed + FlywheelExecutionRateMissing` 收斂為僅剩 `FlywheelExecutionRateMissing`；HostBackupFailed 解除。
+- 188/110 負載回到低檔；K3s node CPU 約 3-6%，KM reembed 未造成主機過載。
+
+**下一步**：
+- 將本次 runtime hotfix 對應的 repo changes 走正式 deploy，避免下一版 image 覆蓋 hotfix。
+- 修 `grist.wooo.work` / `registry.wooo.work` public route 或改 DNS-01 renewal；目前舊 renewal config 已停用以避免 certbot timer 每次失敗。
+
 ## 2026-05-05 | 110 Sentry resource limits persistence gap closed

 **背景**：110 guardrail 告警已清，但主機 load 仍有長尾；統帥擔心 Claude Code 只做 live `docker update`，重建後配置又失效。
@@ -3066,3 +3098,42 @@ C1（evolver 加 YAML_RULE guard）+ C2（seeder SQL `AND status != 'deprecated'
 ```bash
 psql $DATABASE_URL -f apps/api/migrations/cleanup_duplicate_deprecated_playbooks.sql
 ```
+
+---
+
+## 2026-05-05（台北）— 四主機重開機後全站冷啟動救援
+
+**觸發**：110 / 120 / 121 / 188 同時重開機後，多數服務異常；統帥要求先恢復所有網站、主機、核心服務，並建立完整冷啟動 SOP。
+
+### 已恢復
+
+| 範圍 | 結果 |
+|------|------|
+| 188 host PostgreSQL | WAL checkpoint 損壞；已備份後 `pg_resetwal`，`k3s_datastore` `REINDEX` + `VACUUM ANALYZE` 完成 |
+| K3s datastore | 刪除並備份可重建的腐壞 HPA / VPA / VPA checkpoint / `mon1` node rows；120 / 121 重新 Ready |
+| AWOOI prod | `awoooi-api` / `awoooi-web` / `awoooi-worker` Running；VIP `192.168.0.125` 內網驗證 API 200 / Web 307 |
+| mo.wooo.work | `momo-db` WAL redo 損壞；備份後 `pg_resetwal`，`momo-pro-system` / scheduler / bot / DB 全部 healthy；公網 `/` 200、`/health` 200 |
+| 110 host overload | actions runner units 維持最後放行；Sentry ClickHouse/Kafka 已從 dirty-reboot 損壞中恢復，Sentry stack healthy |
+| 188 SignOz | SignOz ClickHouse volume 出現 filesystem corruption；已 clean-clone 可讀資料並保留原始 corrupt volume，SignOz HTTP 恢復 |
+| 冷啟動 SOP | 新增 `docs/runbooks/FULL-STACK-COLD-START-SOP.md` 與 `scripts/reboot-recovery/full-stack-cold-start-check.sh` |
+
+### 驗證
+
+```bash
+bash scripts/reboot-recovery/full-stack-cold-start-check.sh --send-alert-test
+# PASS=31 WARN=0 BLOCKED=0
+# Result: GREEN. Full stack is ready for controlled runner/CD release.
+```
+
+### Dirty reboot 資料保全
+
+- 110 Sentry ClickHouse：原始壞 volume 保留為 `/var/lib/docker/volumes/sentry-clickhouse/_data.corrupt-20260505-203346`；以 clean-clone 恢復可讀資料並加 `force_restore_data`。
+- 110 Sentry Kafka：malformed checkpoint 已備份至 `/var/backups/sentry-kafka-checkpoints-20260505-203942`，只重建 checkpoint，不刪 topic/log data。
+- 188 SignOz ClickHouse：原始壞 volume 保留為 `/var/lib/docker/volumes/signoz-clickhouse/_data.corrupt-20260505-203735`；以 clean-clone 恢復可讀資料。
+- 188 `momo-db`：WAL reset 前備份 `/var/backups/postgresql/momo-db-before-pg-resetwal-20260505-200834.tgz`。
+
+### 已知隔離 / 後續
+
+- 110 actions runner units 仍按策略最後放行：guardrail 已套用，`CPUQuota=200%`、`MemoryMax=2G`、`WatchdogUSec=0`；需在 load/core 穩定後逐步開啟。
+- `Bad message` / `Structure needs cleaning` 是 host filesystem 層訊號；線上 clean-clone 已恢復服務，但完整歷史資料追溯需安排離線 `fsck` 或備份驗證。
+- `drift-scanner-29633040-qrf8w` 為單次 CronJob Error，不阻斷主服務；後續可清理或調查。
--- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md
+++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md
@@ -0,0 +1,497 @@
+# AWOOOI Full-Stack Cold Start SOP
+
+> Version: v1.0
+> Last updated: 2026-05-05 Asia/Taipei
+> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
+
+---
+
+## 0. When To Use This
+
+Use this SOP when any of these happen:
+
+- 110/120/121/188 reboot unexpectedly.
+- All services are abnormal after a power/network event.
+- K3s is stuck `activating`.
+- Host load remains high during startup and service health is mixed.
+- Monitoring, alerting, CD, AI auto-repair, and Docker Compose services disagree about the real state.
+
+The rule is simple: **recover the dependency chain, not the loudest symptom.**
+
+---
+
+## 1. Golden Startup Order
+
+```text
+0. Freeze automation and preserve evidence
+1. Physical/network layer
+2. 188 data layer
+3. 110 registry/observability layer
+4. 120/121 K3s layer
+5. AWOOOI workload layer
+6. Public routes and alert chain
+7. High-load batch/consumer/crawler services
+8. Runner/CD
+9. AI auto-remediation
+10. 112 Kali scanner, if needed
+```
+
+Never start runner/CD before 188 PostgreSQL, 110 Harbor, K3s nodes, and AWOOOI API are healthy.
+
+---
+
+## 2. Automation Freeze
+
+Cold start creates noisy metrics and partial failures. During P0/P1, keep automation in observe-only mode.
+
+| Item | Cold-start policy | Reason |
+|------|-------------------|--------|
+| Gitea/GitHub runners | Last | Build jobs can saturate 110 CPU/RAM. |
+| momo-scheduler / crawlers | Last | Chrome and batch work can saturate 188. |
+| Sentry/Snuba consumers | Controlled | Kafka backlog and ClickHouse merge can create temporary high load. |
+| Alertmanager outbound notification | Gate | Avoid alert storms before API webhook and Telegram are verified. |
+| AI auto-repair | Observe-only | Metrics, Redis, KM, and playbooks may be incomplete. |
+| Stateful DB restart | Human approval | PostgreSQL, Redis, ClickHouse, Harbor DB, Sentry DB are not generic restart targets. |
+
+---
+
+## 3. P0 Evidence And Network
+
+Run from any machine on the same LAN:
+
+```bash
+for h in 110 120 121 188; do
+  ping -c 2 -W 2 192.168.0.$h >/dev/null && echo "PING_OK 192.168.0.$h" || echo "PING_FAIL 192.168.0.$h"
+done
+
+arp -an | grep -E '192\.168\.0\.(110|120|121|188)'
+for h in 110 120 121 188; do
+  nc -G 3 -z 192.168.0.$h 22 && echo "SSH_OK 192.168.0.$h" || echo "SSH_FAIL 192.168.0.$h"
+done
+```
+
+Then capture reboot evidence:
+
+```bash
+ssh ollama@192.168.0.188 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
+ssh wooo@192.168.0.110 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
+ssh wooo@192.168.0.120 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
+ssh wooo@192.168.0.121 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
+```
+
+If any host has ARP `incomplete` or SSH port down, stop here and fix physical/network first.
+
+---
+
+## 4. P0 188 Data Layer
+
+188 is the first real service dependency because K3s datastore and AWOOOI DB depend on PostgreSQL.
+
+### 4.1 Startup order
+
+1. `containerd`
+2. `docker`
+3. `postgresql@14-main`
+4. `k3s_datastore.kine` maintenance
+5. `redis-server` on `6380`
+6. `ollama` or current AI proxy dependencies
+7. `nginx`
+8. Docker networks
+9. MinIO / OpenClaw / SignOz
+10. momo / litellm / batch services after load is stable
+
+### 4.2 Read-only check
+
+```bash
+ssh ollama@192.168.0.188 '
+hostname; date; uptime; free -h
+systemctl is-active containerd docker postgresql@14-main redis-server ollama nginx || true
+pg_isready -h localhost -p 5432 || true
+redis-cli -p 6380 ping 2>/dev/null || redis-cli ping 2>/dev/null || true
+docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | head -120
+'
+```
+
+### 4.3 PostgreSQL WAL checkpoint damage
+
+Signature:
+
+```text
+PANIC: could not locate a valid checkpoint record
+invalid primary checkpoint record
+unexpected pageaddr ... in log segment ...
+```
+
+This blocks:
+
+- `188:5432`
+- K3s startup on 120/121
+- AWOOOI API DB access
+- Alertmanager webhook if API cannot start
+
+Human-approved recovery command on 188:
+
+```bash
+sudo systemctl stop postgresql@14-main
+sudo install -d -m 700 -o postgres -g postgres /var/backups/postgresql
+sudo tar -C /var/lib/postgresql/14 -czf /var/backups/postgresql/14-main-before-pg-resetwal-$(date +%Y%m%d-%H%M%S).tgz main
+sudo -u postgres /usr/lib/postgresql/14/bin/pg_resetwal -f /var/lib/postgresql/14/main
+sudo systemctl start postgresql@14-main
+pg_isready -h localhost -p 5432
+sudo -u postgres psql -d k3s_datastore -c "VACUUM ANALYZE kine;"
+```
+
+Do not run `DROP`, reinitialize the cluster, delete `/var/lib/postgresql`, or restore an old backup unless the commander explicitly approves it.
+
+---
+
+## 5. P0/P1 110 Registry And Observability
+
+110 must recover Harbor/Gitea/Monitoring early, but runners last.
+
+### 5.1 Startup order
+
+1. `docker`
+2. Remove `Exited (128)` / `Exited (137)` orphan containers
+3. Harbor `harbor-log`
+4. Harbor full stack
+5. Gitea
+6. Prometheus / Alertmanager / Grafana / exporters
+7. Langfuse
+8. SignOz
+9. Sentry DB layer
+10. Sentry web/worker/consumer layer
+11. Gitea host runner and actions runners
+
+### 5.2 Checks
+
+```bash
+ssh wooo@192.168.0.110 '
+hostname; date; uptime; free -h
+systemctl is-active docker || true
+curl -s -o /dev/null -w "harbor=%{http_code}\n" --max-time 5 http://127.0.0.1:5000/v2/ || true
+curl -s -o /dev/null -w "gitea=%{http_code}\n" --max-time 5 http://127.0.0.1:3001/ || true
+curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true
+curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true
+curl -s -o /dev/null -w "sentry=%{http_code}\n" --max-time 10 http://127.0.0.1:9000/ || true
+docker ps --format "{{.Names}}\t{{.Status}}" | head -120
+'
+```
+
+Harbor healthy means `/v2/` returns `200` or `401`. Do not treat `401` as failure.
+
+### 5.3 Runner gate
+
+Runner may start only after all are true:
+
+- `188 PostgreSQL` ready
+- `110 Harbor` ready
+- `110 Gitea` ready
+- `120/121 K3s` nodes ready
+- AWOOOI API health passes
+- 110 load/core is below `1.0` for at least 15 minutes
+- runner systemd guardrails are active: `CPUQuota=200%`, `MemoryMax=2G`, `WatchdogUSec=0`
+
+Check:
+
+```bash
+ssh wooo@192.168.0.110 '
+for u in $(systemctl list-units "actions.runner.*" --all --no-legend --plain | awk "{print \$1}"); do
+  echo "=== $u ==="
+  systemctl show "$u" -p ActiveState -p SubState -p CPUQuotaPerSecUSec -p MemoryMax -p WatchdogUSec -p NRestarts
+done
+'
+```
+
+If `WatchdogUSec` is not `0`, apply the guardrail script manually with sudo:
+
+```bash
+sudo /home/wooo/scripts/apply-runner-systemd-guardrails.sh --apply
+```
+
+---
+
+## 6. P1 120/121 K3s
+
+K3s must wait for 188 PostgreSQL and 110 Harbor.
+
+### 6.1 Startup order
+
+1. 120 `k3s.service`
+2. 121 `k3s-agent.service` or its live role
+3. CNI / kube-proxy
+4. Nodes Ready
+5. Core pods
+6. `awoooi-prod` pods
+7. keepalived VIP `192.168.0.125`
+8. NodePorts `32334` and `32335`
+
+### 6.2 Checks
+
+```bash
+ssh wooo@192.168.0.120 '
+hostname; uptime
+pg_isready -h 192.168.0.188 -p 5432 || true
+systemctl is-active k3s k3s-agent keepalived 2>/dev/null || true
+kubectl get nodes -o wide 2>/dev/null || true
+kubectl get pods -A 2>/dev/null | grep -v -E "Running|Completed" || true
+kubectl get pods -n awoooi-prod -o wide 2>/dev/null || true
+ip addr show | grep 192.168.0.125 || true
+'
+
+ssh wooo@192.168.0.121 '
+hostname; uptime
+systemctl is-active k3s k3s-agent keepalived 2>/dev/null || true
+ip addr show | grep 192.168.0.125 || true
+'
+```
+
+If K3s is `activating` while 188 PostgreSQL is down, fix PostgreSQL first. Restarting K3s repeatedly will not solve it.
+
+---
+
+## 7. P2 AWOOOI Workloads
+
+Run after K3s nodes are Ready:
+
+```bash
+ssh wooo@192.168.0.120 '
+kubectl get deploy -n awoooi-prod
+kubectl get pods -n awoooi-prod -o wide
+kubectl get svc -n awoooi-prod
+kubectl get events -n awoooi-prod --sort-by=.lastTimestamp | tail -40
+'
+
+curl -s --max-time 8 http://192.168.0.125:32334/api/v1/health
+curl -s -o /dev/null -w "web=%{http_code}\n" --max-time 8 http://192.168.0.125:32335/
+```
+
+If pods are `ImagePullBackOff`, go back to 110 Harbor.
+
+If API health fails because DB/Redis is down, go back to 188.
+
+---
+
+## 8. P2 Alert Chain
+
+Current main path:
+
+```text
+Prometheus/Alertmanager on 110
+  -> http://192.168.0.125:32334/api/v1/webhooks/alertmanager
+  -> AWOOOI API
+  -> TelegramGateway
+  -> Telegram
+```
+
+Alertmanager health alone is not enough. Run E2E:
+
+```bash
+curl -s -X POST http://192.168.0.125:32334/api/v1/webhooks/alertmanager \
+  -H 'Content-Type: application/json' \
+  -d '{"receiver":"cold-start-test","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"ColdStartE2ETest","severity":"info"},"annotations":{"summary":"Cold start E2E test, ignore"},"startsAt":"2026-05-05T11:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":""}],"groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"cold-start-test"}'
+```
+
+Expected: API returns success and Telegram receives the test alert.
+
+---
+
+## 9. P2 Schedules And Delayed Work
+
+Do not mark the reboot complete until scheduled work is proven runnable. A container can be healthy while its cron path is broken.
+
+| Host / Layer | Required check | Success baseline |
+|--------------|----------------|------------------|
+| 188 cron | `systemctl is-active cron` and `crontab -l` | cron active; backup, restart exporter, stats exporter entries present |
+| 188 backup-from-110 | `backup_110_last_success_timestamp` in textfile/Prometheus | last success age `< 25h` |
+| 188 momo-scheduler | `docker logs momo-scheduler` | `全部排程任務已註冊`; Google Drive auth works; dashboard URLs use container-reachable hostnames |
+| 188 momo import | manual `run_auto_import_task()` after parser changes | selected sheet is `即時業績明細`; imported date range has matching rows in `daily_sales_snapshot` and `realtime_sales_monthly` |
+| 110 cron | `systemctl is-active cron` | cron active; Docker/systemd textfile exporters fresh |
+| 110 startup units | `systemctl --failed` | zero failed units; stale `momo-startup-complete` and `wooo-staggered-startup` disabled |
+| 120 K8s CronJobs | `kubectl get cronjobs -n awoooi-prod` | unsuspended; no failed Jobs remain after current validation |
+| 121 DR drill | `crontab -l` | DR drill cron present unless explicitly paused |
+
+Useful checks:
+
+```bash
+ssh ollama@192.168.0.188 'systemctl is-active cron; crontab -l; ls -l /home/ollama/node_exporter_textfiles/*.prom'
+ssh wooo@192.168.0.110 'systemctl --failed --no-pager; systemctl is-active cron; crontab -l'
+ssh wooo@192.168.0.120 'sudo kubectl get cronjobs,jobs -n awoooi-prod'
+ssh wooo@192.168.0.121 'systemctl is-active cron; crontab -l'
+```
+
+If a schedule succeeds but emits a false verification alert, fix the verification rule before releasing AI auto-remediation. False positives train operators to ignore real alarms.
+
+---
+
+## 10. P2/P3 Stateful Service Guardrails
+
+| Tier | Examples | Automation |
+|------|----------|------------|
+| BLOCK | PostgreSQL data dir, ClickHouse data dir, Harbor DB, Sentry DB | No automatic destructive action. Human approval only. |
+| CRITICAL_HITL | Redis, Kafka, MinIO, SignOz ClickHouse, Sentry ClickHouse | Human-in-the-loop restart/repair. |
+| STANDARD_HITL | API/Web/worker, OpenClaw, litellm | Restart only with evidence and blast-radius check. |
+| AUTO | Stateless exporters, blackbox, nginx exporter | Auto restart allowed after verification. |
+
+Never use generic `docker restart $(docker ps -q)` during cold start.
+
+### 10.1 Dirty-Reboot Storage Corruption
+
+Treat these log signatures as storage corruption, not ordinary service flakiness:
+
+- `Bad message`
+- `Structure needs cleaning`
+- `Unknown codec`
+- `PANIC: could not locate a valid checkpoint record`
+- Kafka `Malformed line` in checkpoint files
+- ClickHouse `broken and needs manual correction`
+
+Cold-start automation may stop a restart storm and collect evidence, but it must not delete the original data directory. If a filesystem returns `Bad message` or `Structure needs cleaning`, the real root cause is below the container layer. Online recovery can restore service from readable data, but complete historical recovery requires an offline filesystem check or backup restore.
+
+### 10.2 ClickHouse Clean-Clone Recovery Pattern
+
+Use this pattern for Sentry ClickHouse or SignOz ClickHouse when individual corrupted parts cannot be moved because the host filesystem rejects reads.
+
+```text
+1. Stop the compose stack or at least stop dependent consumers.
+2. Disable restart loops for the failing container.
+3. Save logs and build an exclude list from unreadable store paths.
+4. Preserve the original volume as _data.corrupt-YYYYMMDD-HHMMSS.
+5. Create a clean _data clone with readable files only.
+6. Add flags/force_restore_data.
+7. Start ClickHouse first, then web/API, then consumers.
+8. Verify HTTP, merge backlog, and restart count before releasing high-load services.
+```
+
+Do not replace this with `rm -rf store/...` unless the unreadable path is already backed up or the commander explicitly accepts data loss. The preferred incident artifact is:
+
+```text
+/var/lib/docker/volumes/<volume>/_data.corrupt-YYYYMMDD-HHMMSS
+/var/backups/<service>-<component>-YYYYMMDD-HHMMSS
+```
+
+### 10.3 Kafka Checkpoint Recovery Pattern
+
+If Kafka refuses to start with malformed checkpoint files after a dirty reboot, preserve and move only checkpoint files:
+
+```text
+log-start-offset-checkpoint
+recovery-point-offset-checkpoint
+replication-offset-checkpoint
+```
+
+Then start Kafka and confirm health before starting Snuba/Sentry consumers. Do not delete topic directories or Kafka logs during cold-start recovery.
+
+---
+
+## 11. P3 High-Load Services
+
+Only release these after P0/P1/P2 gates are green:
+
+| Host | Service | Release condition |
+|------|---------|-------------------|
+| 188 | momo-scheduler / crawler | load/core < 1.0 for 15 minutes and DB healthy |
+| 188 | SignOz ClickHouse | healthy and merge backlog trending down |
+| 188 | litellm | `/health/liveliness` good and provider route verified |
+| 110 | Sentry Snuba consumers | ClickHouse healthy and Kafka backlog decreasing |
+| 110 | Sentry uptime-checker | Sentry web/DB healthy |
+| 110 | runners | all previous gates green and load/core < 1.0 for 15 minutes |
+
+---
+
+## 12. Baseline And AI Auto-Remediation Gate
+
+### 12.1 Stable Runtime Baseline
+
+These are release gates after the first cold-start recovery pass:
+
+| Area | Baseline |
+|------|----------|
+| 188 host | PostgreSQL accepting, Redis PONG, momo `/health` 200, SignOz HTTP reachable, load/core < 1.0 sustained before crawlers |
+| 110 host | Harbor `/v2/` 200/401, Gitea 200/302, Prometheus ready, Alertmanager healthy, Sentry HTTP 200/302/400, no ClickHouse/Kafka restart loop |
+| K3s | 120/121 nodes Ready, VIP `192.168.0.125` present, AWOOOI API 2xx/3xx, Web 2xx/3xx |
+| Public routes | `https://awoooi.wooo.work/api/v1/health` 2xx/3xx, `https://mo.wooo.work/health` 2xx/3xx |
+| Guardrails | Docker/systemd textfile exporters fresh, runner `CPUQuota=200%`, `MemoryMax=2G`, `WatchdogUSec=0` |
+| Schedules | cron active on 110/188/120/121; K8s CronJobs unsuspended; no current failed Jobs; 188 backup success `< 25h` |
+| Backlog | ClickHouse merges and Kafka/Snuba lag trending down, not increasing for two consecutive checks |
+
+If service health is green but load average remains high, check live CPU and IO before changing memory limits. High load after Sentry/Snuba or ClickHouse startup can be backlog drain; high CPU from runners/builds/crawlers is a release-order problem.
+
+### 12.2 AI Auto-Remediation Gate
+
+AI auto-repair can move from observe-only to limited execution only after:
+
+- Prometheus rules are loaded.
+- docker/systemd textfile exporter files are fresh.
+- blackbox probes have stable results.
+- cron/CronJob schedule checks are green.
+- AWOOOI API `/api/v1/health` passes.
+- Alertmanager E2E webhook passes.
+- Redis/KM/playbook health is available.
+- No active restart storm.
+- Host load/core remains below `1.0` for 15 minutes.
+
+Until then:
+
+- diagnose only
+- notify only
+- require human approval for remediation
+- no DB/ClickHouse/Harbor/Sentry destructive action
+- no generic restart action against stateful services
+
+---
+
+## 13. One-Command Readiness Script
+
+Run:
+
+```bash
+bash scripts/reboot-recovery/full-stack-cold-start-check.sh
+```
+
+The script is read-only. It reports gates:
+
+- `P0-NETWORK`
+- `P0-188-DATA`
+- `P0-110-REGISTRY`
+- `P1-K3S`
+- `P2-WORKLOAD`
+- `P2-ALERTCHAIN`
+- `P2-PUBLIC-ROUTES`
+- `P2-SCHEDULES`
+- runner guardrail state inside `P0-110-REGISTRY-OBSERVABILITY`
+
+If it prints `BLOCKED`, fix the first blocked gate before moving forward.
+
+---
+
+## 14. Done Criteria
+
+All must be true:
+
+- Four hosts reachable by SSH.
+- 188 PostgreSQL and Redis healthy.
+- 110 Harbor, Gitea, Prometheus, Alertmanager healthy.
+- 120/121 K3s nodes Ready.
+- VIP `192.168.0.125` present.
+- AWOOOI API and Web reachable through NodePort/VIP.
+- Alertmanager E2E webhook succeeds.
+- cron/CronJob schedules are active, unsuspended, and verified.
+- Sentry and SignOz are either healthy or explicitly in controlled backlog recovery.
+- High-load batch services are capped or delayed.
+- Runners are guarded and released last.
+- AI auto-remediation is not in full execution mode until all gates are green.
+
+---
+
+## 15. Known Drift To Fix After Recovery
+
+These must be cleaned after the incident, not during P0:
+
+- `SERVICE-ENDPOINTS.md` still has old Prometheus/Alertmanager locations.
+- Audit older docs for direct node webhook targets; current main path should be VIP `192.168.0.125:32334`.
+- OpenClaw `8088` vs `8089` must be live-confirmed and normalized.
+- 188 compose paths drift between `/home/ollama/*` and Ansible `/opt/*`.
+- 110 runner docs still mention Docker runner in places; live startup prefers host `gitea-act-runner-host.service`.
+- `scripts/setup-runner-watchdog.sh` conflicts with the 2026-05-05 runner watchdog disablement guardrail.
+- `grist.wooo.work` / `registry.wooo.work` public HTTP/HTTPS currently route to `aiops.wooo.work`; their old 110 certbot renewal configs are disabled until public routing is corrected or DNS-01 renewal is configured.