wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity

docs(ops): record reboot SOP post-CD readback [skip ci]

Browse Source
This commit is contained in:
ogt
2026-06-25 18:30:21 +08:00
parent cde037cdc7
commit 9afc794853
4 changed files with 55 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/LOGBOOK.md
View File

@@ -278,6 +278,29 @@
**邊界**：本輪沒有 SSH、沒有 host write、沒有 Wazuh manager agent registry 驗收、沒有重新註冊 agent、沒有 Wazuh active response、沒有 Kali active scan 或 `/execute`、沒有 Nginx / firewall / Docker / K8s / workflow / secret 變更、沒有 Telegram live send、沒有 SOAR case create、沒有 auto block，也沒有把工作視窗逐字內容放到前台。
## 2026-06-25｜18:23 post-CD deploy storm 收斂與 reboot SOP live readback
**背景**：15:11 之後 `gitea/main` 連續前進，`d2caa4eb` 的 CD `#3340` 被 `d52583d9` 取代，`d52583d9` 的 CD `#3342` 又被 `20c2c81f` 取代，後續又前進到 `aa70835c` 與 deploy marker `2a9e816a`。這是連續 main push / deploy replacement，不是主機重啟 SOP 或服務恢復失敗；最新 production 真相必須以最後 deploy marker、CD 結果、ArgoCD / K3s image readback、public routes、DB / backup / cold-start quick check 一起判斷。
**18:23 live read-only 證據**：
- 最新 deploy marker：`2a9e816a chore(cd): deploy aa70835 [skip ci]`。
- Gitea Actions：`aa70835` 已有 deploy marker `2a9e816a`；前一輪 `code-review.yaml #3346` / `cd.yaml #3345` 已成功；`ansible-lint.yml #3344` 仍 Waiting，屬 runner / label queue，不作為 production deploy 判定。
- ArgoCD：`awoooi-prod` `sync=Synced`、`health=Healthy`；kustomization live tag 顯示 API/Web 為 `aa70835c7177475430479d8ab68621f59ebeb9b0`，18:23 K3s pod readback 顯示 API/Web/Worker Running。
- Public routes：`/api/v1/health`、`/zh-TW/iwooos`、`/zh-TW/governance?tab=automation-inventory`、`https://mo.wooo.work/health`、`https://stock.wooo.work/` 均回 `200`。
- Post-start quick check：`POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`，warning split `SERVICE=0 BOUNDARY=1 EVIDENCE=2`，`RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。
- Delegated cold-start：`PASS=89 WARN=0 BLOCKED=0`，Result `GREEN`。
- MOMO：health `V10.690`，dedicated preflight `PASS=19 WARN=2 BLOCKED=0`；latest job `57 completed`，`DB_DAILY_FRESHNESS 1|2026-06-24`，current-month DB parity `15383/15383` through `2026-06-24`。
- Backup：110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`offsite_fresh=1`，`rclone_gdrive_fresh=1`，`escrow_missing=5`。
- Wazuh / SOC boundary：production `/api/iwooos/wazuh` 與 `/api/v1/iwooos/wazuh` 已是 `200 disabled_waiting_iwooos_wazuh_owner_gate`，但 `configured=false`、manager query accepted `0`、manager registry accepted `0`、runtime gate `0`；這是資安 registry evidence blocker，不是重啟服務 blocker。
- 110 CPU：load around `15.78 / 11.19 / 9.02` during post-CD readback；top CPU belongs to StockPlatform `next build`, StockPlatform headless Chrome smoke groups, and platform services, while AWOOOI CD action container was present only as active CI/deploy load. No process kill, Docker restart, Nginx reload, firewall / K8s write, or Wazuh runtime action was performed.
**判定**：
- 可宣稱：本輪最新 production deploy 後，主機 / K3s / AWOOI routes / MOMO service and data freshness / backup core / offsite checks are service-green for the latest read-only evidence set。
- 不可宣稱：DR complete、credential escrow complete、Wazuh manager registry accepted、Wazuh active response、host write、runtime security gate、或「每次未來重啟一定全綠」。
- SOP 更新方向：連續 main push 造成的 CD cancellation / replacement 必須被記為 `deploy storm`，只看被取消的舊 run 會誤判；最新版本必須以最後 deploy marker + live image + route / DB / backup / cold-start quick check 收斂為準。
**邊界**：本輪為 read-only / docs-only；`sudo -n kubectl` 僅用於 read-only ArgoCD / deployment image readback。沒有讀 secret value、沒有修改 host / Docker / systemd / Nginx / firewall / K8s / ArgoCD / Wazuh runtime。
## 2026-06-25｜15:04 post-start wrapper cold-start WARN 分級修正與 live readback
**背景**：15:02 使用最新 main 版 `post-start-quick-check.sh --no-color` 跑 read-only 驗證時，cold-start 本身是 `PASS=88 WARN=1 BLOCKED=0`，但 wrapper 只看 cold-start exit code，將 WARN-only 誤判成 `BLOCKED`。這會把「rollout / stale warning」放大成服務 blocker，與 SOP 分層判定目標不符。

18
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.54
> Version: v1.55
> Last updated: 2026-06-25 Asia/Taipei
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
@@ -12,22 +12,24 @@
若只是重啟後要快速判斷能不能宣稱恢復，先跑一頁式總檢查：`scripts/reboot-recovery/post-start-quick-check.sh --no-color`，並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為人工 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。
2026-06-25 15:04 post-start wrapper live read-only refresh supersedes the 14:41 wrapper wording. Hosts, routes, K3s, AWOOOI API health, MOMO service health, MOMO business data freshness, backup core/offsite, and core monitoring/exporter surfaces are green for controlled runner/CD release. MOMO is healthy on `V10.681`; latest import job `57` completed cleanly; `MOMO_DAILY_FRESHNESS 1|2026-06-24`; current-month daily snapshot and realtime tables match through `2026-06-24`. `post-start-quick-check.sh` now parses cold-start `PASS / WARN / BLOCKED` summary before classifying exit codes, so WARN-only rollout/stale evidence is no longer inflated into a service blocker. The latest wrapper returns `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED` when service blockers are zero but `escrow_missing=5` remains. Do not turn this into a DR complete or security/runtime acceptance claim. Wazuh host registry acceptance remains outside this SOP lane and is still not complete.
2026-06-25 18:23 post-CD live read-only refresh supersedes the 15:04 wrapper wording. Consecutive main pushes caused older CD runs to be replaced, so the latest production truth is the final deploy marker `2a9e816a chore(cd): deploy aa70835 [skip ci]`, Gitea CD for `aa70835` has produced deploy marker `2a9e816a`, read-only ArgoCD shows `awoooi-prod Synced / Healthy` at revision `2a9e816a9db6e428e1f497c7e4a1759bb2f63d25`; API/Web/Worker live image tag `aa70835c7177475430479d8ab68621f59ebeb9b0`; 18:23 K3s pods are Running and cold-start result is GREEN, and post-start quick check `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`. Hosts, routes, K3s, AWOOOI API health, MOMO service health, MOMO business data freshness, backup core/offsite, and core monitoring/exporter surfaces are green for controlled runner/CD release. MOMO is healthy on `V10.690`; latest import job `57` completed cleanly; `MOMO_DAILY_FRESHNESS 1|2026-06-24`; current-month daily snapshot and realtime tables match through `2026-06-24`. `post-start-quick-check.sh` parses cold-start `PASS / WARN / BLOCKED` summary before classifying exit codes, so WARN-only rollout/stale evidence is no longer inflated into a service blocker. The wrapper returns `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED` when service blockers are zero but `escrow_missing=5` remains. Do not turn this into a DR complete or security/runtime acceptance claim. Wazuh production routes are now `200 disabled_waiting_iwooos_wazuh_owner_gate`, but `configured=false`, manager query accepted `0`, manager registry accepted `0`, and runtime gate `0`; treat Wazuh as a security registry evidence blocker, not a reboot service blocker.
```text
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
Live cold-start read-only check: 2026-06-25 15:04 wrapper delegated cold-start PASS=89 WARN=0 BLOCKED=0, Result=GREEN.
Post-start quick check: 2026-06-25 15:04 PASS=18 WARN=3 BLOCKED=0; warning split SERVICE=0 BOUNDARY=1 EVIDENCE=2; Result=FULL_STACK_GREEN_DR_ESCROW_BLOCKED; exit code 0.
Live cold-start read-only check: 2026-06-25 18:23 wrapper delegated cold-start PASS=89 WARN=0 BLOCKED=0, Result=GREEN.
Post-start quick check: 2026-06-25 18:23 PASS=18 WARN=3 BLOCKED=0; warning split SERVICE=0 BOUNDARY=1 EVIDENCE=2; Result=FULL_STACK_GREEN_DR_ESCROW_BLOCKED; exit code 0.
Repo-side cold-start v1.42+ live read-only run: MOMO source absence / stale data blocker is cleared by import job 57 and `MOMO_DAILY_FRESHNESS 1|2026-06-24`. Live 110 script sync is not claimed until a separate approved deployment/sync happens.
110 live-sync parity: 2026-06-24 23:15 read-only `verify-cold-start-monitor-deploy.sh` correctly BLOCKED because repo script hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`. Do not use live 110 monitor output to prove v1.42 behavior until the approved live-sync gate in §13.3.1 passes.
Service state: FULL_STACK_GREEN_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, public routes/TLS green, MOMO data fresh, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
Runtime release state: API/Web/Worker are ready; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. 14:16 direct route smoke returned 200 for AWOOOI API, `/zh-TW/iwooos`, MOMO health, and Stock; cold-start raw route gate returned all expected route statuses, including redirects such as awoooi web=307 and sentry=302.
MOMO release state: mo.wooo.work health is healthy on version V10.681. `momo-pro-system`, `momo-scheduler`, and `momo-telegram-bot` are healthy; scheduler `RestartCount=0`. 15:04 dedicated preflight returns PASS=19 WARN=2 BLOCKED=0, so retain scheduler fail-closed / notification evidence notes, but no service blocker remains.
Runtime release state: API/Web live image tag is `aa70835c7177475430479d8ab68621f59ebeb9b0`, and 18:23 K3s readback shows API/Web/Worker pods Running; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. 18:23 direct route smoke returned 200 for AWOOOI API, `/zh-TW/iwooos`, `/zh-TW/governance?tab=automation-inventory`, MOMO health, and Stock; cold-start raw route gate returned all expected route statuses, including redirects such as awoooi web=307 and sentry=302.
MOMO release state: mo.wooo.work health is healthy on version V10.690. `momo-pro-system`, `momo-scheduler`, and `momo-telegram-bot` are healthy; scheduler `RestartCount=0`. 18:23 dedicated preflight returns PASS=19 WARN=2 BLOCKED=0, so retain recent container-replace / scheduler fail-closed / notification evidence notes, but no service blocker remains.
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly match through 2026-06-24: `daily_sales_snapshot=109061|2025-07-01|2026-06-24`, `MOMO_MONTHLY_SYNC 15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `MOMO_DAILY_FRESHNESS 1|2026-06-24`. Latest import job is `57 completed|即時業績_當日.xlsx|2026-06-25T13:16:47.359958|2026-06-25T13:18:02.964985|15383|15383|0`.
Google Drive / source-file state: 14:16 cold-start reports `MOMO_GDRIVE_TOKEN_STAT 100000:100000:600 scheduler_uid=100000`. Dedicated preflight confirms host token metadata matches scheduler UID and restrictive mode; container token artifact exists with mode `600`. Token content was not read. Future Drive auth/API failure must still be treated as failed import evidence rather than no-file success.
110 CPU/load readback: 2026-06-25 10:58 user-approved minimal SIGTERM targeted only orphan `stockplatform-review-bulk-ux` Chrome process groups `438005`, `471295`, `640155`, and `670628`; `OLD_GROUPS_REMAINING` returned empty. 15:04 readback shows current higher load is active Gitea Actions / CD `next build` and Playwright e2e smoke, with Chrome as an active child of Playwright, not orphan Chrome. No Docker/systemd/Nginx/firewall/K8s write was performed; do not cancel active CI/smoke unless separately approved.
Backup / monitoring state: 15:04 wrapper readback confirms backup core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09, and escrow_missing=5.
110 CPU/load readback: 2026-06-25 10:58 user-approved minimal SIGTERM targeted only orphan `stockplatform-review-bulk-ux` Chrome process groups `438005`, `471295`, `640155`, and `670628`; `OLD_GROUPS_REMAINING` returned empty. 18:23 readback shows current higher load is mainly StockPlatform `next build`, StockPlatform headless Chrome smoke, and platform services; AWOOOI CD action container was visible only as active CI/deploy load. No Docker/systemd/Nginx/firewall/K8s write was performed; do not cancel active CI/smoke unless separately approved.
Backup / monitoring state: 18:23 wrapper readback confirms backup core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09, and escrow_missing=5.
Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
Deploy storm / CD replacement state: if several main commits land during recovery, older CD runs may be canceled by newer commits. Do not treat the canceled run as a service failure. Wait for the final deploy marker, verify live image tags, ArgoCD health, public routes, DB freshness, backup status, and post-start quick check before declaring latest production recovered.
Wazuh / SOC boundary state: production Wazuh read-only route presence is not equivalent to Wazuh registry recovery. `/api/iwooos/wazuh` and `/api/v1/iwooos/wazuh` returning `200 disabled_waiting_iwooos_wazuh_owner_gate` only proves the route boundary is deployed; manager registry accepted, owner evidence accepted, active response, host write, agent re-enroll, restart, secret patch, Kali active scan, and runtime gate remain `0 / false`.
Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
Allowed declaration: full-stack service readiness is GREEN for controlled runner/CD release; core hosts, routes, K3s, backup/exporter surfaces, AWOOOI API health, MOMO service health, and MOMO data freshness are green for the latest read-only evidence set.
Forbidden declaration: DR complete, credential escrow complete, Wazuh host registry accepted, 110 live monitor synced, or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged.

16
docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md
View File

@@ -183,6 +183,22 @@ CPU / runaway：orphan=? active_ci=? load=?
## 6. 目前最新已驗證基線
2026-06-25 18:23 wrapper live run after deploy marker `2a9e816a`：
- Gitea / CD：`code-review.yaml #3346` success，`cd.yaml #3345` success，deploy marker `2a9e816a chore(cd): deploy aa70835 [skip ci]`。
- GitOps / K3s：ArgoCD `awoooi-prod` `Synced / Healthy`；API/Web/Worker live image tag `aa70835c7177475430479d8ab68621f59ebeb9b0`；API/Web/Worker ready。
- Wrapper：`POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`。
- Warning split：`SERVICE=0 BOUNDARY=1 EVIDENCE=2`。
- Result：`FULL_STACK_GREEN_DR_ESCROW_BLOCKED`，exit code `0`。
- Cold-start：`PASS=89 WARN=0 BLOCKED=0`，Result `GREEN`。
- MOMO：`V10.690`，dedicated preflight `PASS=19 WARN=2 BLOCKED=0`，job `57` clean，`DB_DAILY_FRESHNESS 1|2026-06-24`。
- Backup：110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`。
- Routes：AWOOI API / IwoooS / Governance / MOMO health / Stock all `200` in direct smoke.
- Wazuh / SOC：`/api/iwooos/wazuh` and `/api/v1/iwooos/wazuh` route boundary is `200 disabled_waiting_iwooos_wazuh_owner_gate`; `configured=false`, manager registry accepted `0`, runtime gate `0`. Treat this as a security registry evidence blocker, not a reboot service blocker.
- DR：`escrow_missing=5`，不可宣稱 DR complete。
- CPU：110 load still elevated around `15.78 / 11.19 / 9.02`; current dominant load is StockPlatform `next build`, StockPlatform headless Chrome smoke, and platform services, not an AWOOI service blocker. Do not kill smoke / CI without explicit approval.
- Deploy storm note：`d2caa4eb` and `d52583d9` CD runs were superseded by later main commits; use the latest deploy marker and live image readback, not canceled runs, as production truth.
2026-06-25 15:04 wrapper live run：
- Wrapper：`POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`。

12
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -11,15 +11,15 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-25 15:04 post-start quick check returned exit `0`, `POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`, warning split `SERVICE=0 BOUNDARY=1 EVIDENCE=2`, result `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`: 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOOI API health is healthy/prod/mock=false, delegated cold-start is `PASS=89 WARN=0 BLOCKED=0`, MOMO service health is healthy on `V10.681`, MOMO data freshness is `1|2026-06-24`, 110 / 188 runtime and backup checks are green。MOMO latest valid job `57` completed cleanly at `2026-06-25T13:18:02`, `15383/15383/0`, and current-month snapshot / realtime bounds match through `2026-06-24`. DR remains blocked because credential escrow evidence markers are still missing (`escrow_missing=5`) and must not be forged. |
| Overall recovery readiness | FULL_STACK_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-25 18:23 post-CD quick check returned exit `0`, `POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`, warning split `SERVICE=0 BOUNDARY=1 EVIDENCE=2`, result `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`: latest deploy marker is `2a9e816a chore(cd): deploy aa70835 [skip ci]`; latest main deploy marker `2a9e816a` is present after `aa70835`; read-only ArgoCD shows `awoooi-prod Synced / Healthy` at revision `2a9e816a9db6e428e1f497c7e4a1759bb2f63d25`; API/Web/Worker images are `aa70835c7177475430479d8ab68621f59ebeb9b0`; 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOI API health is healthy/prod/mock=false, delegated cold-start is `PASS=89 WARN=0 BLOCKED=0`, MOMO service health is healthy on `V10.690`, MOMO data freshness is `1|2026-06-24`, 110 / 188 runtime and backup checks are green. MOMO latest valid job `57` completed cleanly at `2026-06-25T13:18:02`, `15383/15383/0`, and current-month snapshot / realtime bounds match through `2026-06-24`. DR remains blocked because credential escrow evidence markers are still missing (`escrow_missing=5`) and must not be forged. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-25 09:05 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, VIP `192.168.0.125` is present, node filesystem / disk-pressure / readonly events are `0`, and latest `km-vectorize-29705460-55rgs` completed. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-25 09:05 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-25 02:35:09`。DR remains blocked on real non-secret credential escrow evidence IDs. |
| P2 service / data truth | GREEN | 100% | Public route/TLS, API/Web route, MOMO health `V10.681`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, direct 15:04 wrapper public route smoke all expected 2xx/3xx, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. 15:04 preflight confirms app / scheduler / Telegram bot healthy, scheduler restart count `0`, token metadata aligned to scheduler UID, latest job `57` completed cleanly, and `DB_DAILY_FRESHNESS 1|2026-06-24`. |
| P3 docs / automation contracts | DONE_WITH_WRAPPER_COLD_START_WARN_CLASSIFIER | 100% | Workplan, SOP v1.54, one-page post-start quick check wrapper + fallback runbook, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO data freshness hard blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, 10:04 scheduler fail-closed live proof, 10:35 route / DB / backup refresh, 11:44 MOMO dedicated preflight blocked readback, 14:16 MOMO dedicated preflight recovery on V10.674 / job 57 / freshness 1, 14:41 wrapper warning split, 15:04 cold-start WARN-only classifier fix, 10:58 user-approved 110 orphan Chrome SIGTERM evidence, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with full cold-start GREEN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
| P2 service / data truth | GREEN | 100% | Public route/TLS, API/Web route, MOMO health `V10.690`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, direct 18:23 wrapper public route smoke all expected 2xx/3xx, current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. 18:23 preflight confirms app / scheduler / Telegram bot healthy, scheduler restart count `0`, token metadata aligned to scheduler UID, latest job `57` completed cleanly, and `DB_DAILY_FRESHNESS 1|2026-06-24`. |
| P3 docs / automation contracts | DONE_WITH_POST_CD_DEPLOY_STORM_READBACK | 100% | Workplan, SOP v1.55, one-page post-start quick check wrapper + fallback runbook, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO data freshness hard blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, 10:04 scheduler fail-closed live proof, 10:35 route / DB / backup refresh, 11:44 MOMO dedicated preflight blocked readback, 14:16 MOMO dedicated preflight recovery on V10.674 / job 57 / freshness 1, 14:41 wrapper warning split, 15:04 cold-start WARN-only classifier fix, 18:23 deploy-storm replacement readback after marker `2a9e816a`, 10:58 user-approved 110 orphan Chrome SIGTERM evidence, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with full cold-start GREEN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
2026-06-25 15:04 supplemental wrapper readback supersedes the 14:41 wrapper wording: direct route smoke is 200 for AWOOOI API / IwoooS / MOMO health / Stock, and cold-start public route/TLS gate is green for all expected 2xx/3xx routes. Repo-side cold-start returns `PASS=89 WARN=0 BLOCKED=0`; `/backup/scripts/backup-status.sh --no-notify --no-refresh` reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO dedicated preflight returns `PASS=19 WARN=2 BLOCKED=0`; MOMO health is `V10.681`; 110 load is around `8.99 / 5.76 / 4.89`, with active Gitea Actions / CD `next build` / Playwright e2e smoke visible, not orphan Chrome. Wrapper result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DEGRADED`, because service warnings are `0` and only DR boundary / evidence warnings remain.
2026-06-25 18:23 post-CD wrapper readback supersedes the 15:04 wording: consecutive main pushes created a deploy storm where `d2caa4eb` CD `#3340` and `d52583d9` CD `#3342` were superseded by later commits. Latest production truth is deploy marker `2a9e816a chore(cd): deploy aa70835 [skip ci]`, Gitea CD for `aa70835` has produced deploy marker `2a9e816a`, ArgoCD `Synced / Healthy`, API/Web/Worker image tag `aa70835c7177475430479d8ab68621f59ebeb9b0`, direct route smoke 200 for AWOOI API / IwoooS / Governance / MOMO health / Stock, and wrapper `POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`. Repo-side cold-start returns `PASS=89 WARN=0 BLOCKED=0`; `/backup/scripts/backup-status.sh --no-notify --no-refresh` reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO dedicated preflight returns `PASS=19 WARN=2 BLOCKED=0`; MOMO health is `V10.690`; 110 load is around `15.78 / 11.19 / 9.02`, with StockPlatform `next build`, StockPlatform headless Chrome smoke, and platform services visible, not an AWOOI service blocker. Wrapper result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DEGRADED`, because service warnings are `0` and only DR boundary / evidence warnings remain. Wazuh route readback is now `200 disabled_waiting_iwooos_wazuh_owner_gate`, but manager registry accepted remains `0`, so Wazuh is a security registry evidence blocker rather than a reboot service blocker.
Full cold-start service readiness may now be declared GREEN for the latest verified evidence set. As of 2026-06-25 15:04, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, MOMO service health is `V10.681`, and MOMO business data is fresh through `2026-06-24`. The live read-only cold-start scorecard is `PASS=89 WARN=0 BLOCKED=0`, and the post-start wrapper result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
Full cold-start service readiness may now be declared GREEN for the latest verified evidence set. As of 2026-06-25 18:23, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, MOMO service health is `V10.690`, and MOMO business data is fresh through `2026-06-24`. The live read-only cold-start scorecard is `PASS=89 WARN=0 BLOCKED=0`, and the post-start wrapper result is `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`. Do not declare DR scorecard complete while credential escrow evidence remains blocked, and do not declare Wazuh registry recovery until manager registry evidence is accepted.
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.
@@ -181,7 +181,7 @@ Next: <single next action>
| P3-005 | DONE | 100 | Update cold-start SOP | SOP now includes start, shutdown, reboot, record, comparison, and 120 blocker handling. | Increment SOP version after each process change. | SOP has controlled power-operation sections and ledger template. |
| P3-006 | DONE | 100 | Update backup status | Backup status now reflects current cron, rclone latest-only, failure-only alert posture, and escrow blocker. | Refresh after 120 backup rerun. | Backup status no longer claims noisy success Telegram notifications. |
| P3-007 | DONE | 100 | Harden Gitea backup stale dump handling | 2026-06-05 manual Gitea backup failed because the container retained `/tmp/gitea-dump.zip` from the 02:00 failure. `scripts/backup/backup-gitea.sh` now renames stale container dump files to timestamped evidence before running a new dump, and the live 110 script is updated. | Watch the next 02:00 Gitea backup. | `bash -n` passes locally and on 110; manual Gitea backup completed after stale evidence rename. |
| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.52 adds one-page post-start quick check wrapper, fallback runbook, startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline, K3s filesystem event blocker, stale-vs-active K8s failed Job classification, post-reboot / post-CD recovery anchors, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note, 110 runaway browser / CI load 分流 PlayBook, healthy-heartbeat suppression, 188 node-exporter restore, 188 DB/Redis exporter restore, 188 MinIO/Velero restore, 188 nginx-exporter restore, 110 Docker disk cleanup boundary, MOMO Google Drive token userns readback, MOMO data freshness hard blocker, post-reboot notification noise gates, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side scorecard source-absence classifier, 110 live-sync parity gate, CD monitoring coverage target-down classification, MOMO dedicated token/source preflight, MOMO V10.674 / StartedAt / lifecycle / job 57 / freshness 1 recovery readback, and 2026-06-25 110 CPU orphan Chrome vs active CI 分流 evidence. | Use `scripts/reboot-recovery/post-start-quick-check.sh --no-color` for T+10 post-reboot triage, then use `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` as manual fallback and SOP v1.52 for exceptions, Plan B, blocker-specific recovery, and historical comparison. Before any real reboot, rerun same-day live cold-start / backup / offsite / alert / escrow / runaway-process / notification-noise / MOMO preflight / monitoring coverage checks. If using the live 110 script, record its hash and do not assume repo-side v1.42 behavior until synced under approval and deploy parity passes. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN`, `K3S_CONTROL_PLANE_AA`, `WORKLOAD_BALANCED`, `B0_ABORTED_BEFORE_REBOOT`, `B1_HOST_RECOVERY_ONLY`, `B2_CORE_SERVICE_READY`, `B3_SERVICE_AVAILABLE_DEGRADED`, `B4_FULL_STACK_GREEN`, and `B5_DR_COMPLETE`; quick check wrapper has one command order and LOGBOOK summary; latest MOMO dedicated preflight returns `PASS=18 WARN=3 BLOCKED=0`; 110 CPU evidence records old orphan Chrome groups removed by approved SIGTERM while active CI load remains observation-only; repeated healthy/same-failure notification noise is controlled without hiding real alerts, and monitoring coverage target-down is routed through exporter restore before any product restart. |
| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.52 adds one-page post-start quick check wrapper, fallback runbook, startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline, K3s filesystem event blocker, stale-vs-active K8s failed Job classification, post-reboot / post-CD recovery anchors, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note, 110 runaway browser / CI load 分流 PlayBook, healthy-heartbeat suppression, 188 node-exporter restore, 188 DB/Redis exporter restore, 188 MinIO/Velero restore, 188 nginx-exporter restore, 110 Docker disk cleanup boundary, MOMO Google Drive token userns readback, MOMO data freshness hard blocker, post-reboot notification noise gates, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side scorecard source-absence classifier, 110 live-sync parity gate, CD monitoring coverage target-down classification, MOMO dedicated token/source preflight, MOMO V10.674 / StartedAt / lifecycle / job 57 / freshness 1 recovery readback, and 2026-06-25 110 CPU orphan Chrome vs active CI 分流 evidence. | Use `scripts/reboot-recovery/post-start-quick-check.sh --no-color` for T+10 post-reboot triage, then use `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` as manual fallback and SOP v1.52 for exceptions, Plan B, blocker-specific recovery, and historical comparison. Before any real reboot, rerun same-day live cold-start / backup / offsite / alert / escrow / runaway-process / notification-noise / MOMO preflight / monitoring coverage checks. If using the live 110 script, record its hash and do not assume repo-side v1.42 behavior until synced under approval and deploy parity passes. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN`, `K3S_CONTROL_PLANE_AA`, `WORKLOAD_BALANCED`, `B0_ABORTED_BEFORE_REBOOT`, `B1_HOST_RECOVERY_ONLY`, `B2_CORE_SERVICE_READY`, `B3_SERVICE_AVAILABLE_DEGRADED`, `B4_FULL_STACK_GREEN`, and `B5_DR_COMPLETE`; quick check wrapper has one command order and LOGBOOK summary; latest MOMO dedicated preflight returns `PASS=19 WARN=2 BLOCKED=0`; 110 CPU evidence records old orphan Chrome groups removed by approved SIGTERM while active CI load remains observation-only; repeated healthy/same-failure notification noise is controlled without hiding real alerts, and monitoring coverage target-down is routed through exporter restore before any product restart. |
| P3-009 | DONE | 100 | Assess 120/121 AA/AS role and host load balancing | 2026-06-12 15:19 live check confirms 120 and 121 are both `Ready control-plane`, `k3s active`, `k3s-agent inactive`, with no taints; however most AWOOOI / ArgoCD / Velero workload remains on 121 after 120 fsck recovery. New assessment defines control-plane AA vs workload AA, migration candidates from 110/188, and stateful migration blockers. | After P0 backup/offsite/cold-start green, implement topology spread for AWOOOI API/Web before moving additional services. | `docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md` exists; SOP v1.6 links AA/AS and load-balancing checks; migration implementation remains explicitly `0%`. |
| P3-010 | DONE | 100 | Update workload balancing docs with 2026-06-13 live truth | Host role assessment, workplan, SOP, backup status, and LOGBOOK are refreshed with current cold-start, backup, 188 certbot degraded, ArgoCD `km-vectorize` degraded, Gitea main `acaae999`, ArgoCD sync, and final pod placement evidence. | Keep updating this file after the next reboot or deploy. | Docs separate service-green status from DR escrow, workload rollout, and non-service governance debt. |
| P3-011 | DONE | 100 | Record `km-vectorize` remediation status | LOGBOOK, this workplan, and SOP now state the schedule/label fix, ArgoCD sync evidence, the invalid manual Job boundary, and the 90% waiting-for-next-schedule gate. | After next 03:00 run, update this row and the top verdict with `lastSuccessfulTime` / ArgoCD health evidence. | No document claims ArgoCD green before official CronJob success evidence exists. |