docs(ops): record 10:35 cold-start freshness readback [skip ci]
This commit is contained in:
ogt
@@ -1,3 +1,22 @@
|
||||
## 2026-06-25｜10:35 live cold-start / route / DB refresh
|
||||
|
||||
**背景**：10:23 已確認 MOMO Drive auth failure 會 fail closed；本輪補一個更近的 read-only refresh，確認 route、backup、DB freshness 與 import job 狀態是否有變。
|
||||
|
||||
**Read-only evidence**：
|
||||
- 110 live cold-start monitor at `2026-06-25 10:35:17 CST` returned expected exit code `2` with `PASS=85 WARN=1 BLOCKED=1`。
|
||||
- Host / K3s：110 / 120 / 121 / 188 ping + SSH OK；K3s `mon` / `mon1` Ready；VIP `192.168.0.125` present；node not-ready / read-only filesystem / disk-pressure / filesystem-error events all `0`。
|
||||
- Public routes direct smoke all returned `200` after redirect-following where applicable：AWOOOI API、`/zh-TW/iwooos`、VibeWork、AwoooGo、MOMO health、Stock、Bitan。
|
||||
- Backup status from 110 remains green for backup surfaces：110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`integrity_stale=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`escrow_missing=5`。
|
||||
- MOMO DB read-only query：`daily_sales_snapshot=104614 rows, 2025-07-01..2026-06-17`；current-month `realtime_sales_monthly=10936 rows, 2026/06/01..2026/06/17`。
|
||||
- Latest import jobs remain unchanged after the fail-closed scheduler proof：job `56 completed` for `即時業績_當日.xlsx`, created `2026-06-18 11:41:00`, completed `2026-06-18 11:42:02`, `10936/10936/0`；jobs `49..53` remain validation failures, not false successes。
|
||||
- MOMO scheduler logs still show the 10:04 Drive auth failure path, Telegram failure notification success, then unrelated price / PChome backfill activity. No newer successful daily-sales import appeared by 10:35。
|
||||
|
||||
**判定**：
|
||||
- 可宣稱：10:35 refresh reconfirms service availability, route health, backup/offsite freshness, and MOMO fail-closed behavior。
|
||||
- 不可宣稱：full-stack green、MOMO data current、Drive token repaired、new daily-sales source imported、DR complete。The hard blocker remains MOMO business data stale beyond 3 days; token metadata/writeback remains a WARN and escrow missing remains a DR blocker。
|
||||
|
||||
**邊界**：本輪全部為 read-only SSH / curl / DB query / log readback；沒有主機寫入、沒有 Docker / Nginx / firewall / K8s / ArgoCD 操作，沒有 Wazuh / 112 / SOC 修改，沒有讀取或保存 secret。
|
||||
|
||||
## 2026-06-25｜10:23 MOMO fail-closed live scheduler proof / cold-start refresh
|
||||
|
||||
**背景**：09:37 已完成 MOMO Drive auth fail-closed 正式部署與 cold-start readback，但當時尚未等到下一次 scheduler 週期證明 live 行為。本輪只做 read-only scheduler log / cold-start / backup evidence refresh，確認程式不再把 Google Drive auth failure 誤報成「沒有新檔案」成功。
|
||||
|
||||
@@ -18,12 +18,13 @@
|
||||
> 2026-06-25 09:05 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; live cold-start is `PASS=87 WARN=1 BLOCKED=1` because MOMO business data is stale and MOMO Google Drive token metadata is missing.
|
||||
> 2026-06-25 09:37 Codex MOMO deploy readback: MOMO `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`, Gitea Actions `cd.yaml #910` succeeded, and 188 host / `momo-scheduler` source now fail closed on Google Drive auth/API failure. Backup/offsite remains green; full-stack still blocks on MOMO data freshness and `escrow_missing=5`.
|
||||
> 2026-06-25 10:23 Codex MOMO fail-closed live proof: 10:04 scheduler run recorded Google Drive auth failure as `❌ 自動匯入失敗` and sent Telegram failure notification successfully; cold-start remains `PASS=87 WARN=1 BLOCKED=1` because business data is stale beyond 3 days and Drive token metadata/writeback is not confirmed. Backup/offsite remains green and `escrow_missing=5` remains the DR blocker.
|
||||
> 2026-06-25 10:35 Codex route / DB / backup refresh: direct public routes for AWOOOI API, IwoooS, VibeWork, AwoooGo, MOMO health, Stock, and Bitan are 200; backup remains 110 `13/13` and 188 `2/2` fresh; MOMO daily and monthly DB bounds still stop at `2026-06-17`; latest import job remains `56 completed`.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-25 10:23 Backup / Offsite / Escrow Live Status
|
||||
## 2026-06-25 10:35 Backup / Offsite / Escrow Live Status
|
||||
|
||||
Read-only evidence sources: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 09:33 Asia/Taipei; scheduler log proof at 10:04; cold-start rerun at 10:23.
|
||||
Read-only evidence sources: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 10:35 Asia/Taipei; scheduler log proof at 10:04; cold-start rerun at 10:35; public route and DB readback at 10:35.
|
||||
|
||||
- 110 backup health: `13/13 fresh failed=0`。
|
||||
- 188 backup health: `2/2 fresh failed=0`。
|
||||
@@ -31,9 +32,10 @@ Read-only evidence sources: `/backup/scripts/backup-status.sh --no-notify --no-r
|
||||
- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。
|
||||
- Last aggregate backup: `2026-06-25 02:35:09`。
|
||||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||||
- Full-stack service release blocker remains separate: cold-start `PASS=87 WARN=1 BLOCKED=1`，原因是 MOMO business data freshness stale (`MOMO_DAILY_FRESHNESS 8|2026-06-17`) plus Google Drive token metadata missing / writeback not confirmed。這不是 backup freshness failure。
|
||||
- Full-stack service release blocker remains separate: cold-start `PASS=85 WARN=1 BLOCKED=1`，原因是 MOMO business data freshness stale (`MOMO_DAILY_FRESHNESS 8|2026-06-17`) plus Google Drive token metadata missing / writeback not confirmed。這不是 backup freshness failure。
|
||||
- MOMO code boundary now covers both failure modes: `cd.yaml #904` makes monthly sync failure fail the import job and prevents Drive file movement; `cd.yaml #910` makes Drive auth/API failure return `success=false` instead of a no-file success.
|
||||
- Live scheduler proof: 2026-06-25 10:04 `auto_import_task` logs `Google Drive 認證失敗: could not locate runnable browser`，then logs `❌ 自動匯入失敗` and sends Telegram failure notification successfully. Therefore the alert is now a correct failure signal, not a heartbeat / no-file false green.
|
||||
- MOMO DB readback: `daily_sales_snapshot=104614|2025-07-01|2026-06-17`; current-month `realtime_sales_monthly=10936|2026/06/01|2026/06/17`; latest import job remains `56 completed` with `10936/10936/0` and no newer successful daily-sales import by 10:35.
|
||||
|
||||
| Gate | Status | Evidence |
|
||||
|------|--------|----------|
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# AWOOOI 全棧冷啟動與主機重啟 SOP
|
||||
|
||||
> Version: v1.47
|
||||
> Version: v1.48
|
||||
> Last updated: 2026-06-25 Asia/Taipei
|
||||
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
|
||||
|
||||
@@ -10,18 +10,18 @@
|
||||
|
||||
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
|
||||
|
||||
2026-06-25 10:23 live read-only refresh supersedes the 09:37 wording. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, and backup/offsite are available, while full-stack release remains blocked by MOMO business data freshness and DR remains blocked by missing credential escrow evidence. MOMO Drive auth false-green is fixed in production by Gitea Actions `cd.yaml #910` at commit `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`: the 10:04 scheduler run proved Google Drive auth failure now records a failed import path and sends a formal failure notification instead of reporting empty-folder success. The Google Drive token ownership/writeback WARN remains because token metadata is still missing; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
|
||||
2026-06-25 10:35 live read-only refresh supersedes the 10:23 wording. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, and backup/offsite are available, while full-stack release remains blocked by MOMO business data freshness and DR remains blocked by missing credential escrow evidence. MOMO Drive auth false-green is fixed in production by Gitea Actions `cd.yaml #910` at commit `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`: the 10:04 scheduler run proved Google Drive auth failure now records a failed import path and sends a formal failure notification instead of reporting empty-folder success, and the 10:35 DB/job readback shows no newer valid daily-sales import has arrived yet. The Google Drive token ownership/writeback WARN remains because token metadata is still missing; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
|
||||
|
||||
```text
|
||||
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
|
||||
Live cold-start read-only check: 2026-06-25 10:23 PASS=87 WARN=1 BLOCKED=1, Result=BLOCKED.
|
||||
Live cold-start read-only check: 2026-06-25 10:35 PASS=85 WARN=1 BLOCKED=1, Result=BLOCKED.
|
||||
Repo-side cold-start v1.42+ live read-only run: MOMO_SOURCE_EMPTY_EVIDENCE_LINES is now 0 because the latest scheduler cycle is a Google Drive auth failure, not a successful empty-folder listing. The hard BLOCKED text is now "188 momo daily sales data stale beyond 3 days". Live 110 script sync is not claimed until a separate approved deployment/sync happens.
|
||||
110 live-sync parity: 2026-06-24 23:15 read-only `verify-cold-start-monitor-deploy.sh` correctly BLOCKED because repo script hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`. Do not use live 110 monitor output to prove v1.42 behavior until the approved live-sync gate in §13.3.1 passes.
|
||||
Service state: SERVICE_AVAILABLE_MOMO_DATA_STALE_GDRIVE_TOKEN_WARN_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
|
||||
Runtime release state: API/Web/Worker are ready; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. `ollama_local` had a short cooldown in the direct health JSON and is not the current release blocker. 09:05 route smoke shows awoooi API=200, `/zh-TW/iwooos`=200, vibework=200, awoooogo=200, momo health=200, stock=200, bitan=200, gitea=200, harbor=200, sentry=200, signoz=200, langfuse=200, aiops=200, registry /v2=401; cold-start raw route gate still records expected redirect statuses such as awoooi web=307 and sentry=302.
|
||||
Runtime release state: API/Web/Worker are ready; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. 10:35 direct route smoke returned 200 for AWOOOI API, `/zh-TW/iwooos`, VibeWork, AwoooGo, MOMO health, Stock, and Bitan. Cold-start raw route gate still records expected redirect statuses such as awoooi web=307 and sentry=302.
|
||||
MOMO release state: mo.wooo.work health is healthy on version V10.657. Gitea main fast-forwarded to e137d7a5d02a7595a44c3f3cc1cf54b766424ee7 and Gitea Actions cd.yaml #910 completed Success for the Drive-auth fail-closed fix. 188 host source and `momo-scheduler` container source both contain the marker `Google Drive 連線或認證失敗，未能確認來源資料夾是否有新檔案`; scheduler / telegram-bot were restarted by CD and are healthy. 10:04 scheduler proof shows auth failure is now logged as `❌ 自動匯入失敗` and Telegram failure notification is sent successfully. Earlier cd.yaml #904 remains the monthly-sync failure boundary fix.
|
||||
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly still match, but both stop at 2026-06-17: `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. `MOMO_DAILY_FRESHNESS 8|2026-06-17` is a hard blocker because business data is not current.
|
||||
Google Drive / source-file state: 2026-06-25 10:23 cold-start reports `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`; direct metadata-only readback confirms host path `/home/ollama/momo-pro/config/google_token.json` is missing and container-side `config/google_token.json` is missing, while the scheduler process runs as UID/GID `100000:100000`. Do not read token content and do not recreate/chown token evidence without an explicit maintenance-window / owner approval. The data blocker is now stale business data with a live auth-failure proof; token missing remains a separate WARN until owner-provided token/writeback evidence is restored. With cd.yaml #910 live, any future Drive auth/API failure must be treated as failed import evidence rather than a no-file success.
|
||||
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly still match, but both stop at 2026-06-17: `daily_sales_snapshot=104614|2025-07-01|2026-06-17`, `realtime_sales_monthly current-month=10936|2026/06/01|2026/06/17`, and cold-start still reports `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. `MOMO_DAILY_FRESHNESS 8|2026-06-17` is a hard blocker because business data is not current. Latest import job remains `56 completed|即時業績_當日.xlsx|2026-06-18 11:41:00|2026-06-18 11:42:02|10936|10936|0`; no newer successful daily-sales import appeared by the 10:35 refresh.
|
||||
Google Drive / source-file state: 2026-06-25 10:35 cold-start reports `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`; direct metadata-only readback confirms host path `/home/ollama/momo-pro/config/google_token.json` is missing and container-side `config/google_token.json` is missing, while the scheduler process runs as UID/GID `100000:100000`. Do not read token content and do not recreate/chown token evidence without an explicit maintenance-window / owner approval. The data blocker is now stale business data with a live auth-failure proof; token missing remains a separate WARN until owner-provided token/writeback evidence is restored. With cd.yaml #910 live, any future Drive auth/API failure must be treated as failed import evidence rather than a no-file success.
|
||||
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09. 09:33 backup-status --no-notify --no-refresh reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, integrity_stale=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5.
|
||||
Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
|
||||
Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
|
||||
|
||||
@@ -11,13 +11,13 @@
|
||||
|
||||
| Area | Status | Completion | Evidence |
|
||||
|------|--------|------------|----------|
|
||||
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_DATA_STALE_GDRIVE_TOKEN_WARN_DR_ESCROW_BLOCKED | 97% | 2026-06-25 10:23 live cold-start returned `PASS=87 WARN=1 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale and Google Drive token ownership/writeback metadata is not confirmed. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOOI API health is healthy/prod/mock=false, MOMO service health is healthy on `V10.657`, 110 / 188 runtime and backup checks are green。MOMO Gitea `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`; `cd.yaml #910` succeeded and deployed a fail-closed Drive auth/API boundary into 188 host source and `momo-scheduler` container source. The 10:04 scheduler cycle proved the live behavior: Google Drive auth failure is now logged as `❌ 自動匯入失敗` and sends Telegram failure notification, not no-file success. Remaining hard service blocker is still MOMO business data freshness: `MOMO_DAILY_FRESHNESS 8|2026-06-17`; DB current-month parity remains `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`; latest valid job `56` is still completed with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Warning evidence: metadata-only check shows `/home/ollama/momo-pro/config/google_token.json` missing on host and `config/google_token.json` missing inside `momo-scheduler`, while scheduler runs as UID/GID `100000:100000`; no token content was read. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
|
||||
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_DATA_STALE_GDRIVE_TOKEN_WARN_DR_ESCROW_BLOCKED | 97% | 2026-06-25 10:35 live cold-start returned `PASS=85 WARN=1 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale and Google Drive token ownership/writeback metadata is not confirmed. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOOI API health is healthy/prod/mock=false, MOMO service health is healthy on `V10.657`, 110 / 188 runtime and backup checks are green。MOMO Gitea `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`; `cd.yaml #910` succeeded and deployed a fail-closed Drive auth/API boundary into 188 host source and `momo-scheduler` container source. The 10:04 scheduler cycle proved the live behavior: Google Drive auth failure is now logged as `❌ 自動匯入失敗` and sends Telegram failure notification, not no-file success. Remaining hard service blocker is still MOMO business data freshness: `MOMO_DAILY_FRESHNESS 8|2026-06-17`; DB current-month readback remains `daily_sales_snapshot=104614|2025-07-01|2026-06-17` and `realtime_sales_monthly=10936|2026/06/01|2026/06/17`; latest valid job `56` is still completed with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Warning evidence: metadata-only check shows `/home/ollama/momo-pro/config/google_token.json` missing on host and `config/google_token.json` missing inside `momo-scheduler`, while scheduler runs as UID/GID `100000:100000`; no token content was read. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
|
||||
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-25 09:05 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, VIP `192.168.0.125` is present, node filesystem / disk-pressure / readonly events are `0`, and latest `km-vectorize-29705460-55rgs` completed. |
|
||||
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-25 09:05 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-25 02:35:09`。DR remains blocked on real non-secret credential escrow evidence IDs. |
|
||||
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS_WITH_GDRIVE_TOKEN_WARN | 98% | Public route/TLS, API/Web route, MOMO health `V10.657`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, 10:04 live scheduler fail-closed proof, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. MOMO latest business date remains `2026-06-17`; stale age is `8` days as of 10:23. Latest valid job `56` already imported `即時業績_當日.xlsx` with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Google Drive token metadata is still a WARN because host and container token paths are missing; this requires owner-gated metadata repair/evidence and must not be solved by reading token contents. |
|
||||
| P3 docs / automation contracts | DONE_WITH_MOMO_FAIL_CLOSED_LIVE_PROOF | 100% | Workplan, SOP v1.47, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, 10:04 scheduler fail-closed live proof, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with Google Drive token metadata WARN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
|
||||
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS_WITH_GDRIVE_TOKEN_WARN | 98% | Public route/TLS, API/Web route, MOMO health `V10.657`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, 10:04 live scheduler fail-closed proof, direct 10:35 public route smoke all 200 for AWOOOI API / IwoooS / VibeWork / AwoooGo / MOMO health / Stock / Bitan, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. MOMO latest business date remains `2026-06-17`; stale age is `8` days as of 10:35. Latest valid job `56` already imported `即時業績_當日.xlsx` with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Google Drive token metadata is still a WARN because host and container token paths are missing; this requires owner-gated metadata repair/evidence and must not be solved by reading token contents. |
|
||||
| P3 docs / automation contracts | DONE_WITH_MOMO_FAIL_CLOSED_LIVE_PROOF | 100% | Workplan, SOP v1.48, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, 10:04 scheduler fail-closed live proof, 10:35 route / DB / backup refresh, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with Google Drive token metadata WARN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
|
||||
|
||||
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-25 10:23, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, and MOMO service health is `V10.657`, but the latest repo-side live read-only cold-start scorecard remains `PASS=87 WARN=1 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and Google Drive token metadata is missing / writeback not confirmed. The hard blocker is now `188 momo daily sales data stale beyond 3 days`; the token state is a separate WARN and not a reason to read token contents. MOMO Drive auth/API failure is no longer allowed to be recorded as a no-file success after CD `#910`; the 10:04 scheduler run proved it now fails closed and sends failure notification. This code fix does not create new business data. This is repo-side source-of-truth evidence and not yet a claim that the 110 live monitor script was deployed. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
|
||||
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-25 10:35, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, and MOMO service health is `V10.657`, but the latest live read-only cold-start scorecard remains `PASS=85 WARN=1 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and Google Drive token metadata is missing / writeback not confirmed. The hard blocker is `188 momo daily sales data stale beyond 3 days`; the token state is a separate WARN and not a reason to read token contents. MOMO Drive auth/API failure is no longer allowed to be recorded as a no-file success after CD `#910`; the 10:04 scheduler run proved it now fails closed and sends failure notification. This code fix does not create new business data. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
|
||||
|
||||
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user