diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 4045325da..3d424e51a 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,31 @@ +## 2026-06-24｜22:17 MOMO import-boundary 正式部署驗證與 full cold-start refresh + +**背景**：21:57 已完成 MOMO import sync failure boundary 的 branch-level 修正與雙機 Codex 基準校準；本輪在使用者批准後，將同一個已測試 commit 走 MOMO `main` / Gitea CD 正式 release lane，並重跑 production / DB / backup / cold-start readback。這是 MOMO release lane 自動部署，不是本視窗手動重啟主機、Docker、Nginx 或 firewall。 + +**Release evidence**： +- MOMO Gitea `main` fast-forward：`8240b59b8422b8bdeda3be8f8942c4eab0bd0634 -> 84035906aba0e5e190d031a13cfd9b47a8cd1f73`。 +- Gitea Actions：`cd.yaml #904` 對應 commit `84035906ab`，狀態 `Success`。 +- Regression before release：`pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` -> `10 passed`。 + +**Production read-only evidence**： +- `https://mo.wooo.work/health` -> `{"database":"postgresql","status":"healthy","version":"V10.653"}`；`https://mo.wooo.work/` redirect-followed -> `200 https://mo.wooo.work/ai_intelligence`。 +- 188 live source marker：`/home/ollama/momo-pro/services/import_service.py` contains `def _table_columns`、`業績分析儀表板同步失敗`、`保留來源檔案等待重試，不移動 Google Drive 檔案`。 +- 188 containers：`momo-pro-system` healthy, `momo-db` up, `momo-scheduler` healthy, `momo-telegram-bot` healthy；scheduler restarted around `2026-06-24 22:13:07` after CD sync mode. +- DB read-only query on `momo_analytics`：`daily_sales_snapshot=104614 rows, 2025/07/01..2026/06/17`；`realtime_sales_monthly=786621 rows, 2024/01/01..2026/06/17`。 +- Recent import jobs remain unchanged: job `56 completed 10936/10936/0`, created `2026-06-18 11:41:00`, completed `2026-06-18 11:42:02`；jobs `52..53` failed by validation, not marked successful. +- User-named routes at 22:16 all returned `200`: `awoooi.wooo.work` -> `/zh-TW`, `vibework.wooo.work`, `awooogo.wooo.work`, `mo.wooo.work` -> `/ai_intelligence`, `stock.wooo.work`, `bitan.wooo.work`。 + +**Full-stack / backup evidence**： +- Full cold-start at `2026-06-24 22:16:12 CST`：`PASS=86 WARN=0 BLOCKED=1`。110 / 120 / 121 / 188 ping + SSH port OK；188 PostgreSQL / Redis / MOMO / exporters OK；110 Harbor / Gitea / Prometheus / Alertmanager / Sentry OK；K3s `mon` / `mon1` Ready；public routes/TLS OK；110 / 188 backup textfiles and storage health OK。 +- Only blocker remains `MOMO_DAILY_FRESHNESS 7|2026-06-17`; `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17` is still green. +- Backup status `--no-notify --no-refresh` at 22:17：110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`integrity_stale=0`，`offsite_fresh=1`，`rclone_gdrive_fresh=1`，`escrow_missing=5`，last backup-all `2026-06-24 02:28:39`。 + +**判定**： +- 可宣稱：MOMO import-boundary fix 已正式部署到 production；下一次若 `realtime_sales_monthly` 同步失敗，job 會 fail 且 Drive 檔不會被移動；核心主機 / K3s / routes / backup / exporters 仍可用。 +- 不可宣稱：MOMO data current、full-stack green、DR complete。MOMO 業務資料仍停在 `2026-06-17`，解除需要合法較新 `即時業績_當日` source file 或 owner-provided source evidence；credential escrow evidence 仍缺 `5`。 + +**邊界**：本輪寫入只有 `git push origin HEAD:main` 觸發既有 MOMO CD release lane，以及 repo docs 更新；本視窗沒有手動 SSH 寫 188、沒有手動 Docker / Nginx / firewall / K3s / ArgoCD runtime 操作，沒有 active scan，沒有讀取或保存 secret，沒有使用聊天中的密碼。 + ## 2026-06-24｜21:57 MOMO import sync failure boundary hardening 與雙機 artifact 校準 **背景**：21:33 readback 已確認主機 / K3s / public routes / backup / offsite 可用，MOMO 程式版本為 `V10.653`，但 business data freshness 仍停在 `2026-06-17`。本輪針對先前 SOP 已列出的高風險路徑做程式層 hardening：`daily_sales_snapshot` 寫入成功但 `realtime_sales_monthly` 同步失敗時，不得把 import job 標成 `completed`，也不得移動 Google Drive 來源檔。 diff --git a/docs/runbooks/BACKUP-STATUS.md b/docs/runbooks/BACKUP-STATUS.md index 492bb6b19..d87f4ed98 100644 --- a/docs/runbooks/BACKUP-STATUS.md +++ b/docs/runbooks/BACKUP-STATUS.md @@ -10,7 +10,32 @@ > 2026-06-13 Codex escrow refresh: 13:10 live report confirms offsite/rclone/script readiness is green and only five non-secret credential escrow evidence markers remain missing. > 2026-06-18 Codex cold-start refresh: full-stack service readiness is green after stale failed Job classification; backup core remains green; DR still blocked only by five credential escrow evidence markers. > 2026-06-24 Codex Velero/exporter refresh: 188 MinIO / Velero backup freshness, 188 PostgreSQL / Redis exporters, 188 node-exporter, and 110 disk pressure are recovered; DR still blocked only by five credential escrow evidence markers and service full-green is blocked by MOMO data freshness. -> 2026-06-24 21:33 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; full-stack remains blocked only by MOMO data freshness. +> 2026-06-24 22:17 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; MOMO import-boundary fix is production-deployed, but full-stack remains blocked by MOMO data freshness. + +--- + +## 2026-06-24 22:17 Backup / Offsite / Escrow Live Status + +Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 22:17 Asia/Taipei. + +- 110 backup health: `13/13 fresh failed=0`。 +- 188 backup health: `2/2 fresh failed=0`。 +- Integrity / configured blockers: `core_blockers=0`、`dr_warnings=5`、`configured_missing_110=0`、`configured_missing_188=0`、`script_missing_110=0`、`script_missing_188=0`。 +- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。 +- Last aggregate backup: `2026-06-24 02:28:39`。 +- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。 +- Full-stack service release blocker remains separate: cold-start `PASS=86 WARN=0 BLOCKED=1`，原因是 `MOMO_DAILY_FRESHNESS 7|2026-06-17`；這不是 backup freshness failure。 +- MOMO code boundary is now production-deployed: Gitea Actions `cd.yaml #904` succeeded at commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`; 188 live source marker confirms monthly sync failure now fails the job and prevents Drive file movement. + +| Gate | Status | Evidence | +|------|--------|----------| +| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0. | +| 188 backup freshness | VERIFIED | 2/2 fresh, failed count 0. | +| Offsite / GDrive freshness | VERIFIED | `offsite_fresh=1`, `rclone_gdrive_fresh=1`. | +| Backup core blockers | GREEN | `core_blockers=0`. | +| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. | +| MOMO code boundary | VERIFIED | `cd.yaml #904` success and 188 live import-service marker. | +| Service full green | NO-GO | Blocked by MOMO data freshness, not by backup. | --- diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 3d6ca1f38..791298f23 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,6 +1,6 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.39 +> Version: v1.40 > Last updated: 2026-06-24 Asia/Taipei > Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path. @@ -10,34 +10,35 @@ 本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。 -2026-06-24 21:57 final readback supersedes the earlier 21:33 wording where it discusses MOMO Codex branch baseline and workstation artifact sync. The service and data readiness gates below are refreshed by the 21:33 live cold-start scorecard plus 21:57 MOMO import-boundary / workstation evidence: +2026-06-24 22:17 final readback supersedes the earlier 21:57 wording where it discusses MOMO import-boundary as branch-only. The service and data readiness gates below are refreshed by the 22:16 live cold-start scorecard, 22:17 backup-status readback, and MOMO Gitea CD `#904` production deploy evidence: ```text Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%. -Live cold-start read-only check: PASS=86 WARN=0 BLOCKED=1, Result=BLOCKED. +Live cold-start read-only check: 2026-06-24 22:16 PASS=86 WARN=0 BLOCKED=1, Result=BLOCKED. Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared. Runtime release state: API/Web/Worker are ready; latest deployment marker 622bc372 points runtime image to 2ec7f6f4 and production API health returns healthy. 21:33 redirect-followed route batch shows awoooi web=200, awoooi API=200, vibework=200, awoooogo=200, momo health=200, stock=200, bitan=200, gitea=200, harbor=200, sentry=200, signoz=200, langfuse=200, registry /v2=401; cold-start raw route gate still records expected redirect statuses such as awoooi web=307, momo web=302, sentry=302. CD #3294 still has a historical Failure record because post-deploy monitoring coverage saw 188 nginx-exporter down before the exporter restore. -MOMO release state: mo.wooo.work health is healthy on version V10.653, matching Gitea main commit 8240b59b8422b8bdeda3be8f8942c4eab0bd0634. Mac Mini and MacBook Pro controlled Codex workspaces are both on branch codex/momo-current-main-dev-base-20260624 at commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73 with dirty=0. This codex branch includes the import-boundary fix where realtime_sales_monthly sync failure marks the job failed and prevents Drive file movement; production deploy still requires the MOMO main/CD lane. -MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly match, but both stop at 2026-06-17. MOMO_DAILY_FRESHNESS is 7 days, which is a hard blocker because business data is not current. +MOMO release state: mo.wooo.work health is healthy on version V10.653. Gitea main fast-forwarded to 84035906aba0e5e190d031a13cfd9b47a8cd1f73 and Gitea Actions cd.yaml #904 completed Success. 188 live source contains the production marker `def _table_columns`, `業績分析儀表板同步失敗`, and `保留來源檔案等待重試，不移動 Google Drive 檔案`, proving the import-boundary fix is deployed. Mac Mini and MacBook Pro controlled Codex workspaces are both on branch codex/momo-current-main-dev-base-20260624 at commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73 with dirty=0. +MOMO data state: full-table read-only DB query shows `daily_sales_snapshot=104614 rows, 2025/07/01..2026/06/17` and `realtime_sales_monthly=786621 rows, 2024/01/01..2026/06/17`. Current-month daily_sales_snapshot and realtime_sales_monthly match, but both stop at 2026-06-17. MOMO_DAILY_FRESHNESS is 7 days, which is a hard blocker because business data is not current. Google Drive / source-file state: momo scheduler token ownership is fixed for Docker userns, container-side Drive listing works, but folder 當日業績匯入 currently has no matching 即時業績_當日 Excel source file. Archive latest matching file is 2026-06-18T01:30:39Z and was already imported by job 56. Mac Mini and MacBook candidate spreadsheets were also read-only inspected: the local current daily candidate only contains 2025-07-01..2025-07-02, the iCloud full-month candidate only contains 2025-06-01..2025-06-30, and MacBook candidates are either header-only or the same 2025-07-01..2025-07-02 dataset. These are not legitimate newer sources. -Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-24 02:28:39, 188 MinIO is healthy, Velero BackupStorageLocation default is Available, one-off backup reboot-recovery-202606240456 completed, backup-health textfile reports Velero freshness green, PostgreSQL / Redis exporters are green, 188 nginx-exporter is restored with nginx_up=1, monitoring coverage is 14/14 jobs UP, and VeleroBackupNotRun / PostgreSQLDown / RedisDown / disk-pressure / nginx-exporter target-down evidence is resolved. 21:33 backup-status no-notify/no-refresh still reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5. +Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-24 02:28:39, 188 MinIO is healthy, Velero BackupStorageLocation default is Available, one-off backup reboot-recovery-202606240456 completed, backup-health textfile reports Velero freshness green, PostgreSQL / Redis exporters are green, 188 nginx-exporter is restored with nginx_up=1, monitoring coverage is 14/14 jobs UP, and VeleroBackupNotRun / PostgreSQLDown / RedisDown / disk-pressure / nginx-exporter target-down evidence is resolved. 22:17 backup-status --no-notify --no-refresh reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, integrity_stale=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5. Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice. Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom. -Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces are recovered; MOMO production code release is current with Gitea main V10.653; both controlled Codex workspaces are aligned on the MOMO import-boundary fix branch; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence. +Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces are recovered; MOMO production code release includes the import-boundary fix at Gitea main 84035906aba0; both controlled Codex workspaces are aligned on the same MOMO fix branch; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence. Forbidden declaration: full-stack green, MOMO data current, DR complete, or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged. ``` -2026-06-24 21:57 Codex workstation continuity readback: +2026-06-24 22:17 Codex workstation continuity readback: ```text MacBook Pro 192.168.0.111 can authenticate to Gitea over SSH with its own public key named MacBook Pro Codex 20260624. MOMO Pro Mac Mini workspace is /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty=0. MOMO Pro MacBook workspace is /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty=0. MOMO import-boundary regression: pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q => 10 passed. -Codex Start Here / workstation dashboard / scorecard safe artifacts were copied to MacBook Pro; remote dashboard readback reports AWOOOI=80604403f316, MOMO=84035906aba0, artifact blocked=0. +MOMO production release: Gitea main and cd.yaml #904 are at 84035906aba0e5e190d031a13cfd9b47a8cd1f73; 188 live source marker proves production deploy. +Codex Start Here / workstation dashboard / scorecard safe artifacts were copied to MacBook Pro; latest artifact dashboard readback is refreshed after the docs closeout commit. Raw Codex App DB, auth, sessions, raw conversations, .env, runtime volumes, raw .git directories, passwords, tokens, and Mac Mini private keys were not copied. AwoooGo MacBook dev workspace remains ready at /Users/ooo/codex-workspaces/awooogo-dev, branch dev, upstream gitea/dev, commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty=0. Safe handoff artifacts still match by local / remote SHA-256 readback after Start Here / workstation dashboard / scorecard refresh. Exact hash values are intentionally not hard-coded in this runbook because they change whenever handoff artifacts are refreshed. Raw Codex App DB, auth, sessions, raw conversations, .env, runtime volumes, raw .git directories, passwords, tokens, and Mac Mini private keys were not copied. -This improves workstation continuity after host reboot / operator relocation, but does not change service cold-start status: full-stack green remains blocked by MOMO data freshness and DR remains blocked by credential escrow evidence. +This improves workstation continuity after host reboot / operator relocation, and the MOMO import-boundary fix is now production-deployed; it does not change service cold-start status: full-stack green remains blocked by MOMO data freshness and DR remains blocked by credential escrow evidence. ``` 2026-06-18 12:17 live readback supersedes older service-availability wording: @@ -216,7 +217,7 @@ DR_COMPLETE = no, because credential escrow evidence is incomplete ```text 110 / 120 / 121 / 188 HOST_READY = yes Core public services SERVICE_READY = yes -MOMO_RELEASE_CURRENT = yes, because mo.wooo.work health is V10.653 and matches Gitea main commit 8240b59b8422b8bdeda3be8f8942c4eab0bd0634 +MOMO_RELEASE_CURRENT = yes, because mo.wooo.work health is V10.653 and Gitea main / CD #904 deployed commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73 MOMO_DB_PARITY = yes MOMO_DATA_FRESH = no, because latest daily_sales_snapshot date is 2026-06-17 and stale age is 7 days as of 2026-06-24 21:04 MOMO_SOURCE_AVAILABLE = no, because Drive intake has no newer 即時業績_當日 source and Mac Mini / MacBook candidate files only contain old or header-only data @@ -230,15 +231,15 @@ DR_COMPLETE = no, because credential escrow evidence is incomplete 重啟後若需要從 Mac Mini / MacBook Pro 繼續 Codex 開發，必須另外確認 Codex safe handoff artifacts，不得把服務恢復與 Codex raw 對話同步混為一談。 -2026-06-24 21:06 Asia/Taipei readback： +2026-06-24 22:17 Asia/Taipei readback： ```text MacBook Pro 192.168.0.111 SSH = OK Safe artifacts synced = Start Here and workstation dashboard readback matched; current SHA-256 values are tracked in the workstation dashboard artifact and local sha256sum readback Start Here readback = registry_ready 3, registry_blocked 8, latest_dev_on_gitea 3, production_on_gitea 8, raw_history_sync False Workstation dashboard readback = artifact_sync_synced 2, artifact_sync_blocked 0, MOMO current main baseline ready 2 -MOMO Pro Mac Mini workspace = /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 8240b59b8422b8bdeda3be8f8942c4eab0bd0634, SYSTEM_VERSION V10.653, dirty 0 -MOMO Pro MacBook workspace = /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 8240b59b8422b8bdeda3be8f8942c4eab0bd0634, SYSTEM_VERSION V10.653, dirty 0 +MOMO Pro Mac Mini workspace = /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty 0 +MOMO Pro MacBook workspace = /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty 0 AwoooGo MacBook workspace = ready on dev commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty 0 ``` @@ -247,6 +248,7 @@ AwoooGo MacBook workspace = ready on dev commit 8471b376d97c1436d4612ece17f51ba0 ```text Mac Mini / MacBook Pro 已同步 Codex 開工入口與治理 snapshot。 MOMO Pro 可以在 Mac Mini / MacBook Pro 從 Gitea current-main Codex baseline 開工；實作前仍需從 codex/momo-current-main-dev-base-20260624 切新的 codex/。 +MOMO import-boundary fix 已經由 main / CD #904 部署到 production；後續仍要等下一個真實匯入檔驗證 failure boundary 是否阻止搬檔。 ``` 禁止宣告： @@ -2027,18 +2029,19 @@ NO-GO: truncate 或 restore 整庫來製造新鮮度。 ### 14.34 2026-06-24 MOMO import sync failure boundary hardening -2026-06-24 21:57 的第八段變更是把 MOMO 自動匯入的「partial success」風險納入重啟 SOP。`daily_sales_snapshot` 寫入成功不代表整體匯入成功；`realtime_sales_monthly` 同步失敗時，必須 fail job、保留來源檔，不得移動 Google Drive 檔案到 archive。 +2026-06-24 21:57 的第八段變更是把 MOMO 自動匯入的「partial success」風險納入重啟 SOP。2026-06-24 22:17 已補正式 release readback：同一修正已 fast-forward 到 MOMO `main`，Gitea Actions `cd.yaml #904` 成功，188 live source marker 已確認。`daily_sales_snapshot` 寫入成功不代表整體匯入成功；`realtime_sales_monthly` 同步失敗時，必須 fail job、保留來源檔，不得移動 Google Drive 檔案到 archive。 -| 項目 | 21:57 MOMO import-boundary baseline | -|------|-------------------------------------| -| SOP version | `v1.39` | +| 項目 | 22:17 MOMO import-boundary production baseline | +|------|------------------------------------------------| +| SOP version | `v1.40` | | Production health | `https://mo.wooo.work/health` healthy, version `V10.653` | -| Live DB read-only | `daily_sales_snapshot=104614 rows, 2025-07-01..2026-06-17`; `realtime_sales_monthly=786621 rows, 2024-01-01..2026-06-17` | +| Live DB read-only | `daily_sales_snapshot=104614 rows, 2025/07/01..2026/06/17`; `realtime_sales_monthly=786621 rows, 2024/01/01..2026/06/17` | | Scheduler read-only | 最近 12 小時 `當日業績匯入` / `即時業績_當日` 均為 `0` 個 Excel，排程不發送成功通知 | | Latest successful import | job `56 completed`, `10936` rows, `2026-06-18 11:41..11:42` | -| Code branch | MOMO `codex/momo-current-main-dev-base-20260624` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73` | +| Code / deploy | MOMO `main` and `codex/momo-current-main-dev-base-20260624` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`; Gitea Actions `cd.yaml #904` Success | +| Live source marker | 188 `/home/ollama/momo-pro/services/import_service.py` contains `_table_columns`, `業績分析儀表板同步失敗`, and `保留來源檔案等待重試，不移動 Google Drive 檔案` | | Regression | `pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` => `10 passed` | -| Production deploy state | Codex branch fix is not a production deploy declaration; MOMO main/CD must carry it before production can be called patched | +| Production deploy state | Production patched for code boundary; data freshness still blocked until a legitimate newer source file imports successfully | MOMO import success 判定： diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index ef9825541..36b06b2d0 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -11,13 +11,13 @@ | Area | Status | Completion | Evidence | |------|--------|------------|----------| -| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED | 98% | 2026-06-24 21:33 live cold-start returned `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, 110 / 188 runtime and backup checks are green。188 `node-exporter`、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO / Velero BSL are restored; monitoring coverage is now `14/14 UP`; 110 disk pressure cleared。Remaining service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive listing works from the scheduler container, but `當日業績匯入` has no newer `即時業績_當日` Excel source file. 2026-06-24 21:57 added MOMO codex branch fix `84035906aba0` so monthly sync failure now fails the import job and prevents Drive file movement; production deploy is still a separate MOMO main/CD gate. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. | +| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED | 98% | 2026-06-24 22:16 live cold-start returned `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, 110 / 188 runtime and backup checks are green。188 `node-exporter`、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO / Velero BSL are restored; monitoring coverage is now `14/14 UP`; 110 disk pressure cleared。Remaining service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive listing works from the scheduler container, but `當日業績匯入` has no newer `即時業績_當日` Excel source file. 2026-06-24 22:17 confirms MOMO `main` and Gitea Actions `cd.yaml #904` deployed `84035906aba0`, so monthly sync failure now fails the import job and prevents Drive file movement in production. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. | | P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. | | P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-24 21:33 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`。188 `node-exporter` textfile scrape、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO endpoint、Velero BSL and latest completed backup freshness are restored; monitoring coverage is `14/14 UP`; `BackupHealthMonitorMissing188`、`PostgreSQLDown`、`RedisDown`、`VeleroBackupNotRun` and 110 disk-pressure alerts resolved. DR remains blocked on real non-secret credential escrow evidence IDs. | -| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 97% | Public route/TLS, API/Web route, momo health `V10.653` matching Gitea main `8240b59b8422b8bdeda3be8f8942c4eab0bd0634`, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. Mac Mini / MacBook Pro controlled MOMO workspaces now both point to codex branch commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`, which fixes monthly sync failure handling, but production deployment of that fix is not claimed. MOMO latest business date remains `2026-06-17`; stale age is `7` days as of 21:57. Drive pending folder has `0` matching files in repeated scheduler checks; Mac Mini / MacBook candidate files are old or header-only, so there is no safe newer source to import. | -| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.39, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MOMO V10.653 / Gitea main / dual-workstation Codex baseline readback, MOMO import-boundary hardening, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. | +| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 98% | Public route/TLS, API/Web route, momo health `V10.653`, MOMO main / CD `#904` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`, 188 live import-boundary source marker, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. Mac Mini / MacBook Pro controlled MOMO workspaces both point to the same codex branch commit. MOMO latest business date remains `2026-06-17`; stale age is `7` days as of 22:17. Drive pending folder has `0` matching files in repeated scheduler checks; Mac Mini / MacBook candidate files are old or header-only, so there is no safe newer source to import. | +| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.40, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MOMO V10.653 / Gitea main / dual-workstation Codex baseline readback, MOMO import-boundary production deploy, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. | -Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-24 21:57, routes/hosts/K3s/backups/exporters/Velero/monitoring coverage are available, and MOMO dev code has a branch-level import-boundary fix, but the latest cold-start scorecard remains `PASS=86 WARN=0 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and no newer legitimate source file is available. Do not declare DR scorecard complete while credential escrow evidence remains blocked. +Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-24 22:17, routes/hosts/K3s/backups/exporters/Velero/monitoring coverage are available, and MOMO production code has the import-boundary fix, but the latest cold-start scorecard remains `PASS=86 WARN=0 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and no newer legitimate source file is available. Do not declare DR scorecard complete while credential escrow evidence remains blocked. 2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback. @@ -160,7 +160,7 @@ Next: | P2-001 | VERIFIED | 100 | Public route smoke | 2026-06-12 18:57 cold-start confirms all listed domains returned expected 2xx/3xx over HTTPS; registry root route returned 200 in the scorecard and `/v2/` remains the normal unauthenticated 401 pattern from earlier checks. This proves ingress/TLS plus current route availability. | Keep as one row in scorecard. | Public route table updated after each reboot. | | P2-002 | BLOCKED_MOMO_DATA_FRESHNESS | 97 | momo latest/current-month parity and freshness | Latest current-month parity is good: `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. Full-table read-only DB evidence is `daily_sales_snapshot=104614 rows, 2025-07-01..2026-06-17` and `realtime_sales_monthly=786621 rows, 2024-01-01..2026-06-17`. However latest business data is stale: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive pending folder `當日業績匯入` has `0` matching `即時業績_當日` Excel files after repeated scheduler checks, and archive latest `2026-06-18T01:30:39Z` was already imported by job `56`. | Wait for or obtain a newer legitimate source file, then verify import job `sync_success=true`, archive movement, table bounds, and `MOMO_DAILY_FRESHNESS <= 2`. | Snapshot/current-month row count and bounds match, source folder has no unprocessed stale file, and daily freshness is within threshold. | | P2-008 | DONE | 100 | Separate MOMO service recovery from upstream source absence | 2026-06-24 11:35 readback proves MOMO service is healthy (`V10.639`), DB parity is good, scheduler container can list Drive, and recent logs have no current token `Permission denied`; the blocker is source-file absence, not service outage. SOP v1.32 records GO/NO-GO rules forbidding old archive re-import, product-export import, truncate, whole-DB restore, or fake freshness. | Keep the stale warning active until a legitimate newer `即時業績_當日` source file appears and imports cleanly. | Operators can say "MOMO service recovered, data pipeline waiting for upstream source file" without calling the full stack green. | -| P2-003 | DONE_DEV_BRANCH_NOT_PROD_DEPLOYED | 96 | Fix momo job semantics | Gitea-first repair is now in `/Users/ogt/codex-workspaces/momo-pro-dev` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73` on branch `codex/momo-current-main-dev-base-20260624`, also fast-forwarded to MacBook Pro. `process_daily_sales_import()` marks monthly sync failure as `failed`, records the sync error in summary, returns `False`, and leaves `auto_import_from_drive()` outside the Drive archive/move path. Regression tests cover both job failure and no-move behavior. | Merge/deploy through the MOMO `main`/CD lane before calling production patched; then watch the next real Google Drive import and confirm no file moves unless both tables sync. | `pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` returns `10 passed`; production closeout requires live deployment readback plus next real import evidence. | +| P2-003 | DONE_PRODUCTION_DEPLOYED_WAITING_NEXT_REAL_IMPORT | 99 | Fix momo job semantics | Gitea-first repair is in `/Users/ogt/codex-workspaces/momo-pro-dev` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73` on branch `codex/momo-current-main-dev-base-20260624`, also fast-forwarded to MacBook Pro and fast-forwarded to MOMO `main`. Gitea Actions `cd.yaml #904` succeeded, and 188 live source contains `_table_columns`, `業績分析儀表板同步失敗`, and `保留來源檔案等待重試，不移動 Google Drive 檔案`. `process_daily_sales_import()` marks monthly sync failure as `failed`, records the sync error in summary, returns `False`, and leaves `auto_import_from_drive()` outside the Drive archive/move path. Regression tests cover both job failure and no-move behavior. | Watch the next real Google Drive import and confirm no file moves unless both tables sync; if a real monthly sync failure happens, verify import job status is `failed` and source file remains pending. | `pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` returns `10 passed`; production deployment/readback is complete; final behavioral closeout requires next real import evidence. | | P2-004 | DONE | 100 | PostgreSQL index corruption runbook path | SOP v1.2 now states `posting list tuple ... cannot be split` is an index repair incident. | Use only concurrent reindex if the error returns. | No truncate, no whole DB restore; `REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;` and idempotent resync evidence recorded. | | P2-005 | VERIFIED | 100 | Do not rely on route 200 only | 2026-06-12 closeout has route + DB + backup + offsite + schedule + alert + K3s + cold-start scorecard evidence. The only remaining blocker is DR credential escrow, outside service availability. | Keep this cross-surface checklist mandatory after every reboot. | Each reboot record has route, DB, backup, schedules, alert, scorecard rows. | | P2-006 | DONE | 100 | Validate momo scheduler WARN | 2026-06-12 post-reboot regression showed the old detector was too narrow for Chinese batch and `[Feeder]` logs. The detector was widened and deployed to 110; 14:47 scorecard reads `SCHEDULER_RECENT_ACTIVITY 1070` and marks scheduler healthy. | Keep normal monitoring; treat future recurrence as detector tuning only if direct logs remain active. | Container healthy, direct log activity exists, and latest scorecard removed this WARN. |