wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

docs(ops): record 11:21 recovery readback [skip ci]

Browse Source
This commit is contained in:
ogt
2026-06-25 11:26:51 +08:00
parent d2854edcd8
commit a24793fee5
3 changed files with 26 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/LOGBOOK.md
View File

@@ -1,3 +1,20 @@
## 2026-06-25｜11:21 CPU 清理後 live route / cold-start / backup refresh
**背景**：11:01 已完成 MOMO preflight gate 與 110 orphan Chrome SIGTERM 收斂；本輪補一個更完整的只讀 refresh，確認服務、route、backup 與 CPU 狀態是否回穩。
**只讀證據**：
- Public routes direct smoke 在必要時 follow redirect 後全部回 `200`：AWOOOI API、`/zh-TW/iwooos`、VibeWork、AwoooGo、MOMO health、Stock、Bitan。
- Repo-side cold-start check 於 `2026-06-25 11:21:07 CST` 回預期 exit code `2`，分數為 `PASS=87 WARN=1 BLOCKED=1`。110 / 120 / 121 / 188 ping + SSH OK；K3s `mon` / `mon1` Ready；VIP present；node storage conditions clean；public routes/TLS green；AWOOOI API 與 Web reachable。
- MOMO 在 cold-start 的狀態未變：`MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`、`MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`、`MOMO_DAILY_FRESHNESS 8|2026-06-17`、latest import job `56 completed` clean。這與 dedicated preflight `PASS=8 WARN=4 BLOCKED=2` 一致。
- 110 的 backup status 使用 `/backup/scripts/backup-status.sh --no-notify --no-refresh` 讀回：核心備份健康但 DR 未完成；110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`integrity_stale=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`escrow_missing=5`、last aggregate `2026-06-25 02:35:09`。
- 110 CPU 在前一輪 orphan cleanup 後，11:20 load 約 `6.71 / 5.79 / 6.00`；`vmstat` 顯示 CPU idle 約 `56%..58%`、iowait `4%`。最高 CPU 來源是有 parent node process 的 active `stockplatform-product-ux-smoke.mjs`，另有 `pnpm install` / `uvicorn`；這不是 orphan Chrome，本輪未清掉。
**判定**：
- 可宣稱：此證據集內 core hosts、K3s、public routes、backup/offsite surfaces、AWOOOI API/Web、MOMO service health、monitoring/exporter surfaces 可用。
- 不可宣稱：full-stack green、MOMO data current、Drive token repaired、DR complete、credential escrow complete。唯一服務硬阻擋仍是 MOMO business data freshness；token metadata 仍是 WARN；credential escrow 仍是 DR blocker。
**邊界**：本輪全部為 read-only SSH / curl / DB / cold-start / backup-status evidence；沒有 Docker / systemd / Nginx / firewall / K8s / ArgoCD write，沒有 CI cancellation，沒有 Wazuh / 112 / SOC change，沒有 secret read。
## 2026-06-25｜11:01 MOMO preflight gate 與 110 CPU orphan Chrome 收斂
**背景**：10:45 已把 MOMO Drive token/source recovery 寫成 owner-gated 文件邊界；同時使用者詢問 110 CPU load 持續偏高。需要把兩者都收斂成可重跑證據：MOMO 不能只看 `/health`，110 CPU 也不能只看 load average。

12
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -10,20 +10,20 @@
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
2026-06-25 10:35 live read-only refresh supersedes the 10:23 wording. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, and backup/offsite are available, while full-stack release remains blocked by MOMO business data freshness and DR remains blocked by missing credential escrow evidence. MOMO Drive auth false-green is fixed in production by Gitea Actions `cd.yaml #910` at commit `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`: the 10:04 scheduler run proved Google Drive auth failure now records a failed import path and sends a formal failure notification instead of reporting empty-folder success, and the 10:35 DB/job readback shows no newer valid daily-sales import has arrived yet. The Google Drive token ownership/writeback WARN remains because token metadata is still missing; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
2026-06-25 11:21 live read-only refresh supersedes the 10:35 / 11:01 wording for service availability. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, backup/offsite, and core monitoring/exporter surfaces are available, while full-stack release remains blocked by MOMO business data freshness and DR remains blocked by missing credential escrow evidence. MOMO Drive auth false-green is fixed in production by Gitea Actions `cd.yaml #910` at commit `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`: the 10:04 scheduler run proved Google Drive auth failure now records a failed import path and sends a formal failure notification instead of reporting empty-folder success, and the 11:21 DB/job readback shows no newer valid daily-sales import has arrived yet. The Google Drive token ownership/writeback WARN remains because token metadata is still missing; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
```text
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
Live cold-start read-only check: 2026-06-25 10:35 PASS=85 WARN=1 BLOCKED=1, Result=BLOCKED.
Live cold-start read-only check: 2026-06-25 11:21 PASS=87 WARN=1 BLOCKED=1, Result=BLOCKED.
Repo-side cold-start v1.42+ live read-only run: MOMO_SOURCE_EMPTY_EVIDENCE_LINES is now 0 because the latest scheduler cycle is a Google Drive auth failure, not a successful empty-folder listing. The hard BLOCKED text is now "188 momo daily sales data stale beyond 3 days". Live 110 script sync is not claimed until a separate approved deployment/sync happens.
110 live-sync parity: 2026-06-24 23:15 read-only `verify-cold-start-monitor-deploy.sh` correctly BLOCKED because repo script hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`. Do not use live 110 monitor output to prove v1.42 behavior until the approved live-sync gate in §13.3.1 passes.
Service state: SERVICE_AVAILABLE_MOMO_DATA_STALE_GDRIVE_TOKEN_WARN_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
Runtime release state: API/Web/Worker are ready; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. 10:35 direct route smoke returned 200 for AWOOOI API, `/zh-TW/iwooos`, VibeWork, AwoooGo, MOMO health, Stock, and Bitan. Cold-start raw route gate still records expected redirect statuses such as awoooi web=307 and sentry=302.
Runtime release state: API/Web/Worker are ready; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. 11:21 direct route smoke returned 200 for AWOOOI API, `/zh-TW/iwooos`, VibeWork, AwoooGo, MOMO health, Stock, and Bitan. Cold-start raw route gate still records expected redirect statuses such as awoooi web=307 and sentry=302.
MOMO release state: mo.wooo.work health is healthy on version V10.657. Gitea main fast-forwarded to e137d7a5d02a7595a44c3f3cc1cf54b766424ee7 and Gitea Actions cd.yaml #910 completed Success for the Drive-auth fail-closed fix. 188 host source and `momo-scheduler` container source both contain the marker `Google Drive 連線或認證失敗，未能確認來源資料夾是否有新檔案`; scheduler / telegram-bot were restarted by CD and are healthy. 10:04 scheduler proof shows auth failure is now logged as `❌ 自動匯入失敗` and Telegram failure notification is sent successfully. Earlier cd.yaml #904 remains the monthly-sync failure boundary fix.
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly still match, but both stop at 2026-06-17: `daily_sales_snapshot=104614|2025-07-01|2026-06-17`, `realtime_sales_monthly current-month=10936|2026/06/01|2026/06/17`, and cold-start still reports `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. `MOMO_DAILY_FRESHNESS 8|2026-06-17` is a hard blocker because business data is not current. Latest import job remains `56 completed|即時業績_當日.xlsx|2026-06-18 11:41:00|2026-06-18 11:42:02|10936|10936|0`; no newer successful daily-sales import appeared by the 10:35 refresh.
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly still match, but both stop at 2026-06-17: `daily_sales_snapshot=104614|2025-07-01|2026-06-17`, `realtime_sales_monthly current-month=10936|2026/06/01|2026/06/17`, and cold-start still reports `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. `MOMO_DAILY_FRESHNESS 8|2026-06-17` is a hard blocker because business data is not current. Latest import job remains `56 completed|即時業績_當日.xlsx|2026-06-18 11:41:00|2026-06-18 11:42:02|10936|10936|0`; no newer successful daily-sales import appeared by the 11:21 refresh.
Google Drive / source-file state: 2026-06-25 10:35 cold-start reports `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`; direct metadata-only readback confirms host path `/home/ollama/momo-pro/config/google_token.json` is missing and container-side `config/google_token.json` is missing, while the scheduler process runs as UID/GID `100000:100000`. Do not read token content and do not recreate/chown token evidence without an explicit maintenance-window / owner approval. The data blocker is now stale business data with a live auth-failure proof; token missing remains a separate WARN until owner-provided token/writeback evidence is restored. With cd.yaml #910 live, any future Drive auth/API failure must be treated as failed import evidence rather than a no-file success. 2026-06-25 11:01 dedicated preflight `scripts/reboot-recovery/momo-drive-token-source-recovery-preflight.sh` returns `PASS=8 WARN=4 BLOCKED=2`: public/local health and scheduler are healthy, token metadata remains missing, and `DB_DAILY_FRESHNESS 8|2026-06-17` remains a hard blocker.
110 CPU/load readback: 2026-06-25 10:58 user-approved minimal SIGTERM targeted only orphan `stockplatform-review-bulk-ux` Chrome process groups `438005`, `471295`, `640155`, and `670628`; `OLD_GROUPS_REMAINING` returned empty. No Docker/systemd/Nginx/firewall/K8s write was performed. Remaining load is attributed to active Gitea Actions / CI build work such as `GITEA-ACTIONS-TASK-7111`, not orphan Chrome; do not cancel CI unless separately approved.
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09. 09:33 backup-status --no-notify --no-refresh reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, integrity_stale=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5.
110 CPU/load readback: 2026-06-25 10:58 user-approved minimal SIGTERM targeted only orphan `stockplatform-review-bulk-ux` Chrome process groups `438005`, `471295`, `640155`, and `670628`; `OLD_GROUPS_REMAINING` returned empty. 11:20 readback shows remaining CPU is active `stockplatform-product-ux-smoke.mjs` with parent node process plus install/build work, not orphan Chrome. No Docker/systemd/Nginx/firewall/K8s write was performed; do not cancel active CI/smoke unless separately approved.
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09. 11:21 backup-status --no-notify --no-refresh reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, integrity_stale=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5.
Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces, AWOOOI API health, and MOMO service health are available for the latest read-only evidence set; MOMO production code includes the monthly-sync failure boundary and Drive-auth fail-closed fixes; the live scheduler has proven auth failure now fails closed and sends a failure notification; MOMO data pipeline remains blocked until valid newer data source / Drive credential evidence is restored through an owner-gated path.

4
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -17,7 +17,9 @@
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS_WITH_GDRIVE_TOKEN_WARN | 98% | Public route/TLS, API/Web route, MOMO health `V10.657`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, 10:04 live scheduler fail-closed proof, direct 10:35 public route smoke all 200 for AWOOOI API / IwoooS / VibeWork / AwoooGo / MOMO health / Stock / Bitan, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. MOMO latest business date remains `2026-06-17`; stale age is `8` days as of 10:35. Latest valid job `56` already imported `即時業績_當日.xlsx` with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Google Drive token metadata is still a WARN because host and container token paths are missing; this requires owner-gated metadata repair/evidence and must not be solved by reading token contents. |
| P3 docs / automation contracts | DONE_WITH_MOMO_PREFLIGHT_AND_CPU_TRIAGE | 100% | Workplan, SOP v1.49, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, 10:04 scheduler fail-closed live proof, 10:35 route / DB / backup refresh, 11:01 MOMO dedicated preflight gate, 10:58 user-approved 110 orphan Chrome SIGTERM evidence, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with Google Drive token metadata WARN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-25 10:35, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, and MOMO service health is `V10.657`, but the latest live read-only cold-start scorecard remains `PASS=85 WARN=1 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and Google Drive token metadata is missing / writeback not confirmed. The hard blocker is `188 momo daily sales data stale beyond 3 days`; the token state is a separate WARN and not a reason to read token contents. MOMO Drive auth/API failure is no longer allowed to be recorded as a no-file success after CD `#910`; the 10:04 scheduler run proved it now fails closed and sends failure notification. This code fix does not create new business data. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
2026-06-25 11:21 supplemental readback supersedes the 10:35 route / DB / backup wording for current evidence: direct route smoke is still 200 for AWOOOI API / IwoooS / VibeWork / AwoooGo / MOMO health / Stock / Bitan; repo-side cold-start returns `PASS=87 WARN=1 BLOCKED=1`; `/backup/scripts/backup-status.sh --no-notify --no-refresh` reports 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO dedicated preflight remains `PASS=8 WARN=4 BLOCKED=2`; 110 CPU is active StockPlatform smoke/build work with parent processes, not orphan Chrome.
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-25 11:21, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, and MOMO service health is `V10.657`, but the latest live read-only cold-start scorecard remains `PASS=87 WARN=1 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and Google Drive token metadata is missing / writeback not confirmed. The hard blocker is `188 momo daily sales data stale beyond 3 days`; the token state is a separate WARN and not a reason to read token contents. MOMO Drive auth/API failure is no longer allowed to be recorded as a no-file success after CD `#910`; the 10:04 scheduler run proved it now fails closed and sends failure notification. This code fix does not create new business data. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.