wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

docs(ops): record momo production import boundary readback [skip ci]

Browse Source
This commit is contained in:
ogt
2026-06-24 22:21:34 +08:00
parent 20cb3e16a7
commit ffc167e282
4 changed files with 83 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -11,13 +11,13 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED | 98% | 2026-06-24 21:33 live cold-start returned `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, 110 / 188 runtime and backup checks are green。188 `node-exporter`、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO / Velero BSL are restored; monitoring coverage is now `14/14 UP`; 110 disk pressure cleared。Remaining service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive listing works from the scheduler container, but `當日業績匯入` has no newer `即時業績_當日` Excel source file. 2026-06-24 21:57 added MOMO codex branch fix `84035906aba0` so monthly sync failure now fails the import job and prevents Drive file movement; production deploy is still a separate MOMO main/CD gate. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED | 98% | 2026-06-24 22:16 live cold-start returned `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, 110 / 188 runtime and backup checks are green。188 `node-exporter`、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO / Velero BSL are restored; monitoring coverage is now `14/14 UP`; 110 disk pressure cleared。Remaining service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive listing works from the scheduler container, but `當日業績匯入` has no newer `即時業績_當日` Excel source file. 2026-06-24 22:17 confirms MOMO `main` and Gitea Actions `cd.yaml #904` deployed `84035906aba0`, so monthly sync failure now fails the import job and prevents Drive file movement in production. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-24 21:33 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`。188 `node-exporter` textfile scrape、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO endpoint、Velero BSL and latest completed backup freshness are restored; monitoring coverage is `14/14 UP`; `BackupHealthMonitorMissing188`、`PostgreSQLDown`、`RedisDown`、`VeleroBackupNotRun` and 110 disk-pressure alerts resolved. DR remains blocked on real non-secret credential escrow evidence IDs. |
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 97% | Public route/TLS, API/Web route, momo health `V10.653` matching Gitea main `8240b59b8422b8bdeda3be8f8942c4eab0bd0634`, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. Mac Mini / MacBook Pro controlled MOMO workspaces now both point to codex branch commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`, which fixes monthly sync failure handling, but production deployment of that fix is not claimed. MOMO latest business date remains `2026-06-17`; stale age is `7` days as of 21:57. Drive pending folder has `0` matching files in repeated scheduler checks; Mac Mini / MacBook candidate files are old or header-only, so there is no safe newer source to import. |
| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.39, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MOMO V10.653 / Gitea main / dual-workstation Codex baseline readback, MOMO import-boundary hardening, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. |
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 98% | Public route/TLS, API/Web route, momo health `V10.653`, MOMO main / CD `#904` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`, 188 live import-boundary source marker, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. Mac Mini / MacBook Pro controlled MOMO workspaces both point to the same codex branch commit. MOMO latest business date remains `2026-06-17`; stale age is `7` days as of 22:17. Drive pending folder has `0` matching files in repeated scheduler checks; Mac Mini / MacBook candidate files are old or header-only, so there is no safe newer source to import. |
| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.40, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MOMO V10.653 / Gitea main / dual-workstation Codex baseline readback, MOMO import-boundary production deploy, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. |
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-24 21:57, routes/hosts/K3s/backups/exporters/Velero/monitoring coverage are available, and MOMO dev code has a branch-level import-boundary fix, but the latest cold-start scorecard remains `PASS=86 WARN=0 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and no newer legitimate source file is available. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-24 22:17, routes/hosts/K3s/backups/exporters/Velero/monitoring coverage are available, and MOMO production code has the import-boundary fix, but the latest cold-start scorecard remains `PASS=86 WARN=0 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and no newer legitimate source file is available. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.
@@ -160,7 +160,7 @@ Next: <single next action>
| P2-001 | VERIFIED | 100 | Public route smoke | 2026-06-12 18:57 cold-start confirms all listed domains returned expected 2xx/3xx over HTTPS; registry root route returned 200 in the scorecard and `/v2/` remains the normal unauthenticated 401 pattern from earlier checks. This proves ingress/TLS plus current route availability. | Keep as one row in scorecard. | Public route table updated after each reboot. |
| P2-002 | BLOCKED_MOMO_DATA_FRESHNESS | 97 | momo latest/current-month parity and freshness | Latest current-month parity is good: `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. Full-table read-only DB evidence is `daily_sales_snapshot=104614 rows, 2025-07-01..2026-06-17` and `realtime_sales_monthly=786621 rows, 2024-01-01..2026-06-17`. However latest business data is stale: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive pending folder `當日業績匯入` has `0` matching `即時業績_當日` Excel files after repeated scheduler checks, and archive latest `2026-06-18T01:30:39Z` was already imported by job `56`. | Wait for or obtain a newer legitimate source file, then verify import job `sync_success=true`, archive movement, table bounds, and `MOMO_DAILY_FRESHNESS <= 2`. | Snapshot/current-month row count and bounds match, source folder has no unprocessed stale file, and daily freshness is within threshold. |
| P2-008 | DONE | 100 | Separate MOMO service recovery from upstream source absence | 2026-06-24 11:35 readback proves MOMO service is healthy (`V10.639`), DB parity is good, scheduler container can list Drive, and recent logs have no current token `Permission denied`; the blocker is source-file absence, not service outage. SOP v1.32 records GO/NO-GO rules forbidding old archive re-import, product-export import, truncate, whole-DB restore, or fake freshness. | Keep the stale warning active until a legitimate newer `即時業績_當日` source file appears and imports cleanly. | Operators can say "MOMO service recovered, data pipeline waiting for upstream source file" without calling the full stack green. |
| P2-003 | DONE_DEV_BRANCH_NOT_PROD_DEPLOYED | 96 | Fix momo job semantics | Gitea-first repair is now in `/Users/ogt/codex-workspaces/momo-pro-dev` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73` on branch `codex/momo-current-main-dev-base-20260624`, also fast-forwarded to MacBook Pro. `process_daily_sales_import()` marks monthly sync failure as `failed`, records the sync error in summary, returns `False`, and leaves `auto_import_from_drive()` outside the Drive archive/move path. Regression tests cover both job failure and no-move behavior. | Merge/deploy through the MOMO `main`/CD lane before calling production patched; then watch the next real Google Drive import and confirm no file moves unless both tables sync. | `pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` returns `10 passed`; production closeout requires live deployment readback plus next real import evidence. |
| P2-003 | DONE_PRODUCTION_DEPLOYED_WAITING_NEXT_REAL_IMPORT | 99 | Fix momo job semantics | Gitea-first repair is in `/Users/ogt/codex-workspaces/momo-pro-dev` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73` on branch `codex/momo-current-main-dev-base-20260624`, also fast-forwarded to MacBook Pro and fast-forwarded to MOMO `main`. Gitea Actions `cd.yaml #904` succeeded, and 188 live source contains `_table_columns`, `業績分析儀表板同步失敗`, and `保留來源檔案等待重試，不移動 Google Drive 檔案`. `process_daily_sales_import()` marks monthly sync failure as `failed`, records the sync error in summary, returns `False`, and leaves `auto_import_from_drive()` outside the Drive archive/move path. Regression tests cover both job failure and no-move behavior. | Watch the next real Google Drive import and confirm no file moves unless both tables sync; if a real monthly sync failure happens, verify import job status is `failed` and source file remains pending. | `pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` returns `10 passed`; production deployment/readback is complete; final behavioral closeout requires next real import evidence. |
| P2-004 | DONE | 100 | PostgreSQL index corruption runbook path | SOP v1.2 now states `posting list tuple ... cannot be split` is an index repair incident. | Use only concurrent reindex if the error returns. | No truncate, no whole DB restore; `REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;` and idempotent resync evidence recorded. |
| P2-005 | VERIFIED | 100 | Do not rely on route 200 only | 2026-06-12 closeout has route + DB + backup + offsite + schedule + alert + K3s + cold-start scorecard evidence. The only remaining blocker is DR credential escrow, outside service availability. | Keep this cross-surface checklist mandatory after every reboot. | Each reboot record has route, DB, backup, schedules, alert, scorecard rows. |
| P2-006 | DONE | 100 | Validate momo scheduler WARN | 2026-06-12 post-reboot regression showed the old detector was too narrow for Chinese batch and `[Feeder]` logs. The detector was widened and deployed to 110; 14:47 scorecard reads `SCHEDULER_RECENT_ACTIVITY 1070` and marks scheduler healthy. | Keep normal monitoring; treat future recurrence as detector tuning only if direct logs remain active. | Container healthy, direct log activity exists, and latest scorecard removed this WARN. |