diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index b0a154663..4045325da 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,30 @@ +## 2026-06-24｜21:57 MOMO import sync failure boundary hardening 與雙機 artifact 校準 + +**背景**：21:33 readback 已確認主機 / K3s / public routes / backup / offsite 可用，MOMO 程式版本為 `V10.653`，但 business data freshness 仍停在 `2026-06-17`。本輪針對先前 SOP 已列出的高風險路徑做程式層 hardening：`daily_sales_snapshot` 寫入成功但 `realtime_sales_monthly` 同步失敗時，不得把 import job 標成 `completed`，也不得移動 Google Drive 來源檔。 + +**Read-only live evidence**： +- 188 `momo-pro-system` / `momo-scheduler` / `momo-db` 均 running，`https://mo.wooo.work/health` 仍回 healthy / PostgreSQL / `V10.653`。 +- 188 `momo-scheduler` 最近 12 小時多次執行 AutoImport；`當日業績匯入` folder 對 pattern `即時業績_當日` 回 `0` 個 Excel，排程因此記錄「沒有找到待匯入的檔案」且不發送成功通知。 +- 容器內 SQLAlchemy read-only DB query：`daily_sales_snapshot=104614 rows, 2025-07-01..2026-06-17`；`realtime_sales_monthly=786621 rows, 2024-01-01..2026-06-17`。 +- 最近 import job：`56 completed`，`10936` rows，created `2026-06-18 11:41:00`，completed `2026-06-18 11:42:02`；jobs `49..53` 是欄位驗證失敗，沒有被標成功。 + +**MOMO code fix / dev baseline**： +- `/Users/ogt/codex-workspaces/momo-pro-dev` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`：`fix(import): fail daily sales job on monthly sync failure`。 +- 修正點：`process_daily_sales_import()` 在 `realtime_sales_monthly` 同步或驗證失敗時，會寫入 summary、標記 job `failed`、`progress=95`、`current_step=業績分析儀表板同步失敗`，回傳 `False`；外層 `auto_import_from_drive()` 因此不會進入 archive / move_file 路徑。 +- Regression tests：`pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` → `10 passed`。 +- Gitea branch：`codex/momo-current-main-dev-base-20260624` 已推到 `84035906aba0e5e190d031a13cfd9b47a8cd1f73`；MacBook Pro `/Users/ooo/codex-workspaces/momo-pro-dev` 已 fast-forward 到同 commit，dirty `0`。 + +**Codex workstation artifact**： +- `~/.codex/CODEX-START-HERE.md` 新增「2026-06-24 21:57 最新恢復錨點」。 +- `~/.codex/codex-workstation-sync-dashboard.snapshot.json` 新增 `current_recovery_readback`：AWOOOI `80604403f316`、MOMO `84035906aba0`、`artifact_sync.blocked_target_count=0`。 +- Safe artifacts 已同步到 MacBook Pro；remote readback：`REMOTE_DASHBOARD_AWOOOI=80604403f316`、`REMOTE_DASHBOARD_MOMO=84035906aba0`、`REMOTE_ARTIFACT_BLOCKED=0`。仍未同步 raw Codex conversations / sessions / auth / secrets。 + +**判定**： +- 可宣稱：MOMO dev baseline 已修掉「月表同步失敗仍 completed / 搬檔」錯誤路徑；Mac Mini / MacBook Pro MOMO Codex 開工基準已對齊。 +- 不可宣稱：MOMO production 已部署此修正、MOMO data current、full-stack green、DR complete。解除 data freshness 仍需新的合法 `即時業績_當日` source file、import job 成功、`sync_success=true`、兩表日期範圍一致且 `MOMO_DAILY_FRESHNESS <= 2`。 + +**邊界**：本輪 live 操作為 read-only DB / scheduler / container evidence；程式修改只在 MOMO codex branch；沒有 SSH 寫 188、沒有 Docker / Nginx / firewall / K3s / ArgoCD runtime 寫入，沒有 active scan，沒有讀取或保存 secret，沒有使用聊天中的密碼。 + ## 2026-06-24｜21:33 recovery refresh、MOMO V10.653 雙機基準與 Wazuh 分工邊界 **背景**：21:18 已確認主機 / K3s / routes / backup 可用，但 MOMO release 又前進到 `V10.653`，且 IwoooS 主控視窗同步 Wazuh 只讀 API 邊界仍在獨立 release lane。這輪只做主機 / route / backup / cold-start / MOMO source truth / Codex workstation 的 read-only 與 docs-only 收斂，不處理 Wazuh runtime。 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 3ecc9c2f1..3d6ca1f38 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,6 +1,6 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.38 +> Version: v1.39 > Last updated: 2026-06-24 Asia/Taipei > Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path. @@ -10,29 +10,31 @@ 本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。 -2026-06-24 21:33 final readback supersedes the earlier 21:18 wording where it discusses public route / backup / MOMO release / MacBook artifact sync freshness. The service and data readiness gates below are refreshed by the 21:33 live cold-start scorecard plus 21:33 source/workstation evidence: +2026-06-24 21:57 final readback supersedes the earlier 21:33 wording where it discusses MOMO Codex branch baseline and workstation artifact sync. The service and data readiness gates below are refreshed by the 21:33 live cold-start scorecard plus 21:57 MOMO import-boundary / workstation evidence: ```text Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%. Live cold-start read-only check: PASS=86 WARN=0 BLOCKED=1, Result=BLOCKED. Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared. Runtime release state: API/Web/Worker are ready; latest deployment marker 622bc372 points runtime image to 2ec7f6f4 and production API health returns healthy. 21:33 redirect-followed route batch shows awoooi web=200, awoooi API=200, vibework=200, awoooogo=200, momo health=200, stock=200, bitan=200, gitea=200, harbor=200, sentry=200, signoz=200, langfuse=200, registry /v2=401; cold-start raw route gate still records expected redirect statuses such as awoooi web=307, momo web=302, sentry=302. CD #3294 still has a historical Failure record because post-deploy monitoring coverage saw 188 nginx-exporter down before the exporter restore. -MOMO release state: mo.wooo.work health is healthy on version V10.653, matching Gitea main commit 8240b59b8422b8bdeda3be8f8942c4eab0bd0634. Mac Mini and MacBook Pro controlled Codex workspaces are both on branch codex/momo-current-main-dev-base-20260624 at commit 8240b59b8422b8bdeda3be8f8942c4eab0bd0634 with dirty=0. +MOMO release state: mo.wooo.work health is healthy on version V10.653, matching Gitea main commit 8240b59b8422b8bdeda3be8f8942c4eab0bd0634. Mac Mini and MacBook Pro controlled Codex workspaces are both on branch codex/momo-current-main-dev-base-20260624 at commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73 with dirty=0. This codex branch includes the import-boundary fix where realtime_sales_monthly sync failure marks the job failed and prevents Drive file movement; production deploy still requires the MOMO main/CD lane. MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly match, but both stop at 2026-06-17. MOMO_DAILY_FRESHNESS is 7 days, which is a hard blocker because business data is not current. Google Drive / source-file state: momo scheduler token ownership is fixed for Docker userns, container-side Drive listing works, but folder 當日業績匯入 currently has no matching 即時業績_當日 Excel source file. Archive latest matching file is 2026-06-18T01:30:39Z and was already imported by job 56. Mac Mini and MacBook candidate spreadsheets were also read-only inspected: the local current daily candidate only contains 2025-07-01..2025-07-02, the iCloud full-month candidate only contains 2025-06-01..2025-06-30, and MacBook candidates are either header-only or the same 2025-07-01..2025-07-02 dataset. These are not legitimate newer sources. Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-24 02:28:39, 188 MinIO is healthy, Velero BackupStorageLocation default is Available, one-off backup reboot-recovery-202606240456 completed, backup-health textfile reports Velero freshness green, PostgreSQL / Redis exporters are green, 188 nginx-exporter is restored with nginx_up=1, monitoring coverage is 14/14 jobs UP, and VeleroBackupNotRun / PostgreSQLDown / RedisDown / disk-pressure / nginx-exporter target-down evidence is resolved. 21:33 backup-status no-notify/no-refresh still reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5. Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice. Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom. -Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces are recovered; MOMO code release is current with Gitea main V10.653 and both controlled Codex workspaces are aligned; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence. +Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces are recovered; MOMO production code release is current with Gitea main V10.653; both controlled Codex workspaces are aligned on the MOMO import-boundary fix branch; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence. Forbidden declaration: full-stack green, MOMO data current, DR complete, or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged. ``` -2026-06-24 21:33 Codex workstation continuity readback: +2026-06-24 21:57 Codex workstation continuity readback: ```text MacBook Pro 192.168.0.111 can authenticate to Gitea over SSH with its own public key named MacBook Pro Codex 20260624. -MOMO Pro Mac Mini workspace is /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 8240b59b8422b8bdeda3be8f8942c4eab0bd0634, SYSTEM_VERSION V10.653, dirty=0. -MOMO Pro MacBook workspace is /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 8240b59b8422b8bdeda3be8f8942c4eab0bd0634, SYSTEM_VERSION V10.653, dirty=0. +MOMO Pro Mac Mini workspace is /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty=0. +MOMO Pro MacBook workspace is /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty=0. +MOMO import-boundary regression: pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q => 10 passed. +Codex Start Here / workstation dashboard / scorecard safe artifacts were copied to MacBook Pro; remote dashboard readback reports AWOOOI=80604403f316, MOMO=84035906aba0, artifact blocked=0. AwoooGo MacBook dev workspace remains ready at /Users/ooo/codex-workspaces/awooogo-dev, branch dev, upstream gitea/dev, commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty=0. Safe handoff artifacts still match by local / remote SHA-256 readback after Start Here / workstation dashboard / scorecard refresh. Exact hash values are intentionally not hard-coded in this runbook because they change whenever handoff artifacts are refreshed. Raw Codex App DB, auth, sessions, raw conversations, .env, runtime volumes, raw .git directories, passwords, tokens, and Mac Mini private keys were not copied. This improves workstation continuity after host reboot / operator relocation, but does not change service cold-start status: full-stack green remains blocked by MOMO data freshness and DR remains blocked by credential escrow evidence. @@ -2023,6 +2025,39 @@ NO-GO: 把 current-month parity 當成 data freshness。 NO-GO: truncate 或 restore 整庫來製造新鮮度。 ``` +### 14.34 2026-06-24 MOMO import sync failure boundary hardening + +2026-06-24 21:57 的第八段變更是把 MOMO 自動匯入的「partial success」風險納入重啟 SOP。`daily_sales_snapshot` 寫入成功不代表整體匯入成功；`realtime_sales_monthly` 同步失敗時，必須 fail job、保留來源檔，不得移動 Google Drive 檔案到 archive。 + +| 項目 | 21:57 MOMO import-boundary baseline | +|------|-------------------------------------| +| SOP version | `v1.39` | +| Production health | `https://mo.wooo.work/health` healthy, version `V10.653` | +| Live DB read-only | `daily_sales_snapshot=104614 rows, 2025-07-01..2026-06-17`; `realtime_sales_monthly=786621 rows, 2024-01-01..2026-06-17` | +| Scheduler read-only | 最近 12 小時 `當日業績匯入` / `即時業績_當日` 均為 `0` 個 Excel，排程不發送成功通知 | +| Latest successful import | job `56 completed`, `10936` rows, `2026-06-18 11:41..11:42` | +| Code branch | MOMO `codex/momo-current-main-dev-base-20260624` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73` | +| Regression | `pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` => `10 passed` | +| Production deploy state | Codex branch fix is not a production deploy declaration; MOMO main/CD must carry it before production can be called patched | + +MOMO import success 判定： + +```text +GO: process_daily_sales_import returns True only if daily_sales_snapshot write and realtime_sales_monthly sync / verification both pass. +GO: auto_import_from_drive may move the Drive source file only after process_daily_sales_import returns True. +NO-GO: mark import_jobs.status=completed when sync_success=false. +NO-GO: move or archive the Drive source file when realtime_sales_monthly sync failed. +NO-GO: send a generic success notification for file_count > 0 before verify_import_data_sync passes. +``` + +重啟後若 MOMO data freshness blocked，先分成三層，不要混在一起： + +```text +1. Service availability: /health, container, DB connection. +2. Source availability: Drive pending folder has a legitimate new 即時業績_當日 source file. +3. Data correctness: import job completed with sync_success=true, and daily_sales_snapshot / realtime_sales_monthly match the imported date range. +``` + ### 14.22 重啟後時間軸驗證 每次重啟後照時間軸推進，不要等到最後才一次判定。 diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index ddd07ca4b..ef9825541 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -11,13 +11,13 @@ | Area | Status | Completion | Evidence | |------|--------|------------|----------| -| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED | 98% | 2026-06-24 21:33 live cold-start returned `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, 110 / 188 runtime and backup checks are green。188 `node-exporter`、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO / Velero BSL are restored; monitoring coverage is now `14/14 UP`; 110 disk pressure cleared。Remaining service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive listing works from the scheduler container, but `當日業績匯入` has no newer `即時業績_當日` Excel source file. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. | +| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED | 98% | 2026-06-24 21:33 live cold-start returned `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, 110 / 188 runtime and backup checks are green。188 `node-exporter`、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO / Velero BSL are restored; monitoring coverage is now `14/14 UP`; 110 disk pressure cleared。Remaining service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive listing works from the scheduler container, but `當日業績匯入` has no newer `即時業績_當日` Excel source file. 2026-06-24 21:57 added MOMO codex branch fix `84035906aba0` so monthly sync failure now fails the import job and prevents Drive file movement; production deploy is still a separate MOMO main/CD gate. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. | | P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. | | P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-24 21:33 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`。188 `node-exporter` textfile scrape、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO endpoint、Velero BSL and latest completed backup freshness are restored; monitoring coverage is `14/14 UP`; `BackupHealthMonitorMissing188`、`PostgreSQLDown`、`RedisDown`、`VeleroBackupNotRun` and 110 disk-pressure alerts resolved. DR remains blocked on real non-secret credential escrow evidence IDs. | -| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 96% | Public route/TLS, API/Web route, momo health `V10.653` matching Gitea main `8240b59b8422b8bdeda3be8f8942c4eab0bd0634`, Mac Mini / MacBook Pro controlled MOMO workspaces both on that baseline, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. However MOMO latest business date is `2026-06-17`; stale age is `7` days as of 21:33. Drive pending folder has `0` matching files in repeated scheduler checks; Mac Mini / MacBook candidate files are old or header-only, so there is no safe newer source to import. | -| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.38, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MOMO V10.653 / Gitea main / dual-workstation Codex baseline readback, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. | +| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 97% | Public route/TLS, API/Web route, momo health `V10.653` matching Gitea main `8240b59b8422b8bdeda3be8f8942c4eab0bd0634`, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. Mac Mini / MacBook Pro controlled MOMO workspaces now both point to codex branch commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`, which fixes monthly sync failure handling, but production deployment of that fix is not claimed. MOMO latest business date remains `2026-06-17`; stale age is `7` days as of 21:57. Drive pending folder has `0` matching files in repeated scheduler checks; Mac Mini / MacBook candidate files are old or header-only, so there is no safe newer source to import. | +| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.39, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MOMO V10.653 / Gitea main / dual-workstation Codex baseline readback, MOMO import-boundary hardening, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. | -Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-24 21:33, routes/hosts/K3s/backups/exporters/Velero/monitoring coverage are available, but the latest cold-start scorecard remains `PASS=86 WARN=0 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and no newer legitimate source file is available. Do not declare DR scorecard complete while credential escrow evidence remains blocked. +Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-24 21:57, routes/hosts/K3s/backups/exporters/Velero/monitoring coverage are available, and MOMO dev code has a branch-level import-boundary fix, but the latest cold-start scorecard remains `PASS=86 WARN=0 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and no newer legitimate source file is available. Do not declare DR scorecard complete while credential escrow evidence remains blocked. 2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback. @@ -158,9 +158,9 @@ Next: | ID | Status | % | Work item | Fine analysis | Next action | Done criteria | |----|--------|---:|-----------|---------------|-------------|---------------| | P2-001 | VERIFIED | 100 | Public route smoke | 2026-06-12 18:57 cold-start confirms all listed domains returned expected 2xx/3xx over HTTPS; registry root route returned 200 in the scorecard and `/v2/` remains the normal unauthenticated 401 pattern from earlier checks. This proves ingress/TLS plus current route availability. | Keep as one row in scorecard. | Public route table updated after each reboot. | -| P2-002 | BLOCKED_MOMO_DATA_FRESHNESS | 96 | momo latest/current-month parity and freshness | Latest current-month parity is good: `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. However latest business data is stale: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive pending folder `當日業績匯入` has `0` matching `即時業績_當日` Excel files after token owner repair, and archive latest `2026-06-18T01:30:39Z` was already imported by job `56`. | Wait for or obtain a newer legitimate source file, then verify import job `sync_success=true`, archive movement, table bounds, and `MOMO_DAILY_FRESHNESS <= 2`. | Snapshot/current-month row count and bounds match, source folder has no unprocessed stale file, and daily freshness is within threshold. | +| P2-002 | BLOCKED_MOMO_DATA_FRESHNESS | 97 | momo latest/current-month parity and freshness | Latest current-month parity is good: `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. Full-table read-only DB evidence is `daily_sales_snapshot=104614 rows, 2025-07-01..2026-06-17` and `realtime_sales_monthly=786621 rows, 2024-01-01..2026-06-17`. However latest business data is stale: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive pending folder `當日業績匯入` has `0` matching `即時業績_當日` Excel files after repeated scheduler checks, and archive latest `2026-06-18T01:30:39Z` was already imported by job `56`. | Wait for or obtain a newer legitimate source file, then verify import job `sync_success=true`, archive movement, table bounds, and `MOMO_DAILY_FRESHNESS <= 2`. | Snapshot/current-month row count and bounds match, source folder has no unprocessed stale file, and daily freshness is within threshold. | | P2-008 | DONE | 100 | Separate MOMO service recovery from upstream source absence | 2026-06-24 11:35 readback proves MOMO service is healthy (`V10.639`), DB parity is good, scheduler container can list Drive, and recent logs have no current token `Permission denied`; the blocker is source-file absence, not service outage. SOP v1.32 records GO/NO-GO rules forbidding old archive re-import, product-export import, truncate, whole-DB restore, or fake freshness. | Keep the stale warning active until a legitimate newer `即時業績_當日` source file appears and imports cleanly. | Operators can say "MOMO service recovered, data pipeline waiting for upstream source file" without calling the full stack green. | -| P2-003 | VERIFIED | 95 | Fix momo job semantics | `/Users/ogt/momo-pro-system/services/import_service.py` and live `/home/ollama/momo-pro/services/import_service.py` now mark monthly sync failure as `failed`, write `drive_file_movable=false`, return `False`, emit a failure alert path, and make auto-import aggregate failures as `success=false`. Live 188 backup: `services/import_service.py.bak.20260604-152827`; live hash after patch: `3fc45671986fa4cc155119f588bc1ebefd272927730052e42e2b9eb4352b2586`. | Watch the next real Google Drive import and confirm no file moves unless both tables sync; keep canonical source-control reconciliation open as a separate supply-chain task. | Live isolated temp-DB/real-Excel test passes; containers reloaded healthy; Telegram token/chat markers are present without exposing secrets; latest DB parity remains 404/404. | +| P2-003 | DONE_DEV_BRANCH_NOT_PROD_DEPLOYED | 96 | Fix momo job semantics | Gitea-first repair is now in `/Users/ogt/codex-workspaces/momo-pro-dev` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73` on branch `codex/momo-current-main-dev-base-20260624`, also fast-forwarded to MacBook Pro. `process_daily_sales_import()` marks monthly sync failure as `failed`, records the sync error in summary, returns `False`, and leaves `auto_import_from_drive()` outside the Drive archive/move path. Regression tests cover both job failure and no-move behavior. | Merge/deploy through the MOMO `main`/CD lane before calling production patched; then watch the next real Google Drive import and confirm no file moves unless both tables sync. | `pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` returns `10 passed`; production closeout requires live deployment readback plus next real import evidence. | | P2-004 | DONE | 100 | PostgreSQL index corruption runbook path | SOP v1.2 now states `posting list tuple ... cannot be split` is an index repair incident. | Use only concurrent reindex if the error returns. | No truncate, no whole DB restore; `REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;` and idempotent resync evidence recorded. | | P2-005 | VERIFIED | 100 | Do not rely on route 200 only | 2026-06-12 closeout has route + DB + backup + offsite + schedule + alert + K3s + cold-start scorecard evidence. The only remaining blocker is DR credential escrow, outside service availability. | Keep this cross-surface checklist mandatory after every reboot. | Each reboot record has route, DB, backup, schedules, alert, scorecard rows. | | P2-006 | DONE | 100 | Validate momo scheduler WARN | 2026-06-12 post-reboot regression showed the old detector was too narrow for Chinese batch and `[Feeder]` logs. The detector was widened and deployed to 110; 14:47 scorecard reads `SCHEDULER_RECENT_ACTIVITY 1070` and marks scheduler healthy. | Keep normal monitoring; treat future recurrence as detector tuning only if direct logs remain active. | Container healthy, direct log activity exists, and latest scorecard removed this WARN. |