docs(ops): record momo fail-closed scheduler proof [skip ci]
This commit is contained in:
ogt
@@ -1,3 +1,23 @@
|
||||
## 2026-06-25｜10:23 MOMO fail-closed live scheduler proof / cold-start refresh
|
||||
|
||||
**背景**：09:37 已完成 MOMO Drive auth fail-closed 正式部署與 cold-start readback，但當時尚未等到下一次 scheduler 週期證明 live 行為。本輪只做 read-only scheduler log / cold-start / backup evidence refresh，確認程式不再把 Google Drive auth failure 誤報成「沒有新檔案」成功。
|
||||
|
||||
**Read-only evidence**：
|
||||
- 188 `momo-scheduler` 在 `2026-06-25 10:04:39` 執行 `auto_import_task`，先記錄啟動 Google Drive 自動匯入，接著偵測到舊版 `token.pickle`，嘗試認證後回 `Google Drive 認證失敗: could not locate runnable browser`。
|
||||
- 同一輪 scheduler 已走 fail-closed 路徑：`Google Drive 連線或認證失敗，未能確認來源資料夾是否有新檔案：could not locate runnable browser`，並記錄 `[Scheduler] [AutoImport] ❌ 自動匯入失敗`。
|
||||
- 同一輪已送出正式失敗通知：`準備發送自動匯入失敗通知...`、`Telegram 通知發送成功`、`匯入失敗通知已發送`。這是正確紅燈，不是心跳噪音，也不是 no-file success。
|
||||
- 10:23 full cold-start read-only check returned expected exit code `2` with `PASS=87 WARN=1 BLOCKED=1`。110 / 120 / 121 / 188 reachable；K3s `mon` / `mon1` Ready；VIP present；public routes / AWOOOI API / MOMO service health / backup exporters available。
|
||||
- MOMO current-month DB parity still matches but is stale：`MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`；`MOMO_DAILY_FRESHNESS 8|2026-06-17`。
|
||||
- 10:23 cold-start now reports `MOMO_SOURCE_EMPTY_EVIDENCE_LINES 0`，because the latest scheduler evidence is auth failure rather than a successful empty-folder listing。Hard blocker wording is now `188 momo daily sales data stale beyond 3 days`，and WARN remains `188 momo Google Drive token ownership/writeback not confirmed` with `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`。
|
||||
- Backup status remains separate and green for core backup：110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`integrity_stale=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`escrow_missing=5`。
|
||||
|
||||
**判定**：
|
||||
- 可宣稱：MOMO Drive auth/API failure false-green 已完成 production 行為驗證；scheduler 會 fail job path / 發正式失敗通知，而不是把 auth failure 當「沒有新檔案」成功。
|
||||
- 可宣稱：核心主機、K3s、public routes、AWOOI API health、MOMO service health、backup/offsite surfaces are available for this read-only evidence set。
|
||||
- 不可宣稱：full-stack green、MOMO data current、Google Drive token repaired、new source file imported、credential escrow complete、DR complete。現在剩下的是資料來源 / token metadata owner evidence gate，而不是服務未啟動。
|
||||
|
||||
**邊界**：本輪沒有主機寫入、沒有 Docker / Nginx / firewall / K8s / ArgoCD 操作、沒有 Wazuh / 112 / SOC 修改，沒有讀取或保存 secret，沒有使用聊天中的密碼。09:37 前一輪 MOMO `main` fast-forward / Gitea CD `#910` 是正式 release lane；10:23 本輪只是 read-only evidence refresh。
|
||||
|
||||
## 2026-06-25｜09:37 MOMO Drive auth fail-closed deploy / cold-start readback
|
||||
|
||||
**背景**：09:05 live refresh 顯示 MOMO 資料 freshness 仍停在 `2026-06-17`，且 Google Drive token metadata 缺失。後續 read-only log 追查確認 2026-06-25 00:37 後 scheduler 因缺 JSON token 嘗試 browser OAuth，無頭環境回 `could not locate runnable browser`，但舊程式仍把它折成「沒有找到待匯入檔案」成功。這會讓排程假綠，必須先修 code boundary。
|
||||
|
||||
@@ -17,12 +17,13 @@
|
||||
> 2026-06-24 23:33 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; live cold-start still blocks only on MOMO source absence / data freshness, not backup.
|
||||
> 2026-06-25 09:05 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; live cold-start is `PASS=87 WARN=1 BLOCKED=1` because MOMO business data is stale and MOMO Google Drive token metadata is missing.
|
||||
> 2026-06-25 09:37 Codex MOMO deploy readback: MOMO `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`, Gitea Actions `cd.yaml #910` succeeded, and 188 host / `momo-scheduler` source now fail closed on Google Drive auth/API failure. Backup/offsite remains green; full-stack still blocks on MOMO data freshness and `escrow_missing=5`.
|
||||
> 2026-06-25 10:23 Codex MOMO fail-closed live proof: 10:04 scheduler run recorded Google Drive auth failure as `❌ 自動匯入失敗` and sent Telegram failure notification successfully; cold-start remains `PASS=87 WARN=1 BLOCKED=1` because business data is stale beyond 3 days and Drive token metadata/writeback is not confirmed. Backup/offsite remains green and `escrow_missing=5` remains the DR blocker.
|
||||
|
||||
---
|
||||
|
||||
## 2026-06-25 09:37 Backup / Offsite / Escrow Live Status
|
||||
## 2026-06-25 10:23 Backup / Offsite / Escrow Live Status
|
||||
|
||||
Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 09:33 Asia/Taipei; cold-start rerun at 09:37.
|
||||
Read-only evidence sources: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 09:33 Asia/Taipei; scheduler log proof at 10:04; cold-start rerun at 10:23.
|
||||
|
||||
- 110 backup health: `13/13 fresh failed=0`。
|
||||
- 188 backup health: `2/2 fresh failed=0`。
|
||||
@@ -32,6 +33,7 @@ Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` f
|
||||
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
|
||||
- Full-stack service release blocker remains separate: cold-start `PASS=87 WARN=1 BLOCKED=1`，原因是 MOMO business data freshness stale (`MOMO_DAILY_FRESHNESS 8|2026-06-17`) plus Google Drive token metadata missing / writeback not confirmed。這不是 backup freshness failure。
|
||||
- MOMO code boundary now covers both failure modes: `cd.yaml #904` makes monthly sync failure fail the import job and prevents Drive file movement; `cd.yaml #910` makes Drive auth/API failure return `success=false` instead of a no-file success.
|
||||
- Live scheduler proof: 2026-06-25 10:04 `auto_import_task` logs `Google Drive 認證失敗: could not locate runnable browser`，then logs `❌ 自動匯入失敗` and sends Telegram failure notification successfully. Therefore the alert is now a correct failure signal, not a heartbeat / no-file false green.
|
||||
|
||||
| Gate | Status | Evidence |
|
||||
|------|--------|----------|
|
||||
@@ -41,7 +43,7 @@ Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` f
|
||||
| Backup core blockers | GREEN | `core_blockers=0`. |
|
||||
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
|
||||
| MOMO Drive token metadata | WARN | Host and container token metadata paths are missing; no token content was read. |
|
||||
| MOMO Drive auth false-green | FIX_DEPLOYED | Gitea Actions `cd.yaml #910` success; 188 host and scheduler container source include fail-closed marker. |
|
||||
| MOMO Drive auth false-green | FIX_DEPLOYED_AND_LIVE_PROVEN | Gitea Actions `cd.yaml #910` success; 188 host and scheduler container source include fail-closed marker; 10:04 scheduler cycle failed closed and sent failure notification. |
|
||||
| Service full green | NO-GO | Blocked by MOMO source absence / data freshness; token metadata warning also requires owner-gated evidence. |
|
||||
|
||||
---
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# AWOOOI 全棧冷啟動與主機重啟 SOP
|
||||
|
||||
> Version: v1.46
|
||||
> Version: v1.47
|
||||
> Last updated: 2026-06-25 Asia/Taipei
|
||||
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
|
||||
|
||||
@@ -10,22 +10,22 @@
|
||||
|
||||
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
|
||||
|
||||
2026-06-25 09:37 live read-only refresh supersedes the 09:05 wording. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, and backup/offsite are available, while full-stack release remains blocked by MOMO source absence / business data freshness and DR remains blocked by missing credential escrow evidence. MOMO Drive auth false-green is now fixed in production by Gitea Actions `cd.yaml #910` at commit `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`: auth/API failure now fails closed instead of being reported as an empty folder. The Google Drive token ownership/writeback WARN remains because token metadata is still missing; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
|
||||
2026-06-25 10:23 live read-only refresh supersedes the 09:37 wording. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, and backup/offsite are available, while full-stack release remains blocked by MOMO business data freshness and DR remains blocked by missing credential escrow evidence. MOMO Drive auth false-green is fixed in production by Gitea Actions `cd.yaml #910` at commit `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`: the 10:04 scheduler run proved Google Drive auth failure now records a failed import path and sends a formal failure notification instead of reporting empty-folder success. The Google Drive token ownership/writeback WARN remains because token metadata is still missing; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
|
||||
|
||||
```text
|
||||
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
|
||||
Live cold-start read-only check: 2026-06-25 09:37 PASS=87 WARN=1 BLOCKED=1, Result=BLOCKED.
|
||||
Repo-side cold-start v1.42 live read-only run: New MOMO fields remain MOMO_SOURCE_EMPTY_EVIDENCE_LINES=21, MOMO_IMPORT_CONFIG=當日業績匯入|即時業績_當日, MOMO_LATEST_IMPORT_JOB=56|completed|即時業績_當日.xlsx|2026-06-18T11:41:00.853176|2026-06-18T11:42:02.309425|10936|10936|0. The only BLOCKED text is "188 momo source file absent while daily sales data stale". Live 110 script sync is not claimed until a separate approved deployment/sync happens.
|
||||
Live cold-start read-only check: 2026-06-25 10:23 PASS=87 WARN=1 BLOCKED=1, Result=BLOCKED.
|
||||
Repo-side cold-start v1.42+ live read-only run: MOMO_SOURCE_EMPTY_EVIDENCE_LINES is now 0 because the latest scheduler cycle is a Google Drive auth failure, not a successful empty-folder listing. The hard BLOCKED text is now "188 momo daily sales data stale beyond 3 days". Live 110 script sync is not claimed until a separate approved deployment/sync happens.
|
||||
110 live-sync parity: 2026-06-24 23:15 read-only `verify-cold-start-monitor-deploy.sh` correctly BLOCKED because repo script hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`. Do not use live 110 monitor output to prove v1.42 behavior until the approved live-sync gate in §13.3.1 passes.
|
||||
Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
|
||||
Service state: SERVICE_AVAILABLE_MOMO_DATA_STALE_GDRIVE_TOKEN_WARN_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
|
||||
Runtime release state: API/Web/Worker are ready; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. `ollama_local` had a short cooldown in the direct health JSON and is not the current release blocker. 09:05 route smoke shows awoooi API=200, `/zh-TW/iwooos`=200, vibework=200, awoooogo=200, momo health=200, stock=200, bitan=200, gitea=200, harbor=200, sentry=200, signoz=200, langfuse=200, aiops=200, registry /v2=401; cold-start raw route gate still records expected redirect statuses such as awoooi web=307 and sentry=302.
|
||||
MOMO release state: mo.wooo.work health is healthy on version V10.657. Gitea main fast-forwarded to e137d7a5d02a7595a44c3f3cc1cf54b766424ee7 and Gitea Actions cd.yaml #910 completed Success for the Drive-auth fail-closed fix. 188 host source and `momo-scheduler` container source both contain the marker `Google Drive 連線或認證失敗，未能確認來源資料夾是否有新檔案`; scheduler / telegram-bot were restarted by CD and are healthy. Earlier cd.yaml #904 remains the monthly-sync failure boundary fix.
|
||||
MOMO release state: mo.wooo.work health is healthy on version V10.657. Gitea main fast-forwarded to e137d7a5d02a7595a44c3f3cc1cf54b766424ee7 and Gitea Actions cd.yaml #910 completed Success for the Drive-auth fail-closed fix. 188 host source and `momo-scheduler` container source both contain the marker `Google Drive 連線或認證失敗，未能確認來源資料夾是否有新檔案`; scheduler / telegram-bot were restarted by CD and are healthy. 10:04 scheduler proof shows auth failure is now logged as `❌ 自動匯入失敗` and Telegram failure notification is sent successfully. Earlier cd.yaml #904 remains the monthly-sync failure boundary fix.
|
||||
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly still match, but both stop at 2026-06-17: `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. `MOMO_DAILY_FRESHNESS 8|2026-06-17` is a hard blocker because business data is not current.
|
||||
Google Drive / source-file state: 2026-06-25 09:37 cold-start reports `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`; direct metadata-only readback confirms host path `/home/ollama/momo-pro/config/google_token.json` is missing and container-side `config/google_token.json` is missing, while the scheduler process runs as UID/GID `100000:100000`. Do not read token content and do not recreate/chown token evidence without an explicit maintenance-window / owner approval. The data blocker remains source absence / stale business data; the token missing state is a separate WARN until owner-provided token/writeback evidence is restored. With cd.yaml #910 live, any future Drive auth/API failure must be treated as failed import evidence rather than a no-file success.
|
||||
Google Drive / source-file state: 2026-06-25 10:23 cold-start reports `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`; direct metadata-only readback confirms host path `/home/ollama/momo-pro/config/google_token.json` is missing and container-side `config/google_token.json` is missing, while the scheduler process runs as UID/GID `100000:100000`. Do not read token content and do not recreate/chown token evidence without an explicit maintenance-window / owner approval. The data blocker is now stale business data with a live auth-failure proof; token missing remains a separate WARN until owner-provided token/writeback evidence is restored. With cd.yaml #910 live, any future Drive auth/API failure must be treated as failed import evidence rather than a no-file success.
|
||||
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09. 09:33 backup-status --no-notify --no-refresh reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, integrity_stale=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5.
|
||||
Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
|
||||
Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
|
||||
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces, AWOOOI API health, and MOMO service health are available for the latest read-only evidence set; MOMO production code includes the monthly-sync failure boundary and Drive-auth fail-closed fixes; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence; MOMO Google Drive token/writeback evidence needs a separate non-secret owner-gated repair.
|
||||
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces, AWOOOI API health, and MOMO service health are available for the latest read-only evidence set; MOMO production code includes the monthly-sync failure boundary and Drive-auth fail-closed fixes; the live scheduler has proven auth failure now fails closed and sends a failure notification; MOMO data pipeline remains blocked until valid newer data source / Drive credential evidence is restored through an owner-gated path.
|
||||
Forbidden declaration: full-stack green, MOMO data current, DR complete, credential escrow complete, 110 live monitor synced, or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged.
|
||||
```
|
||||
|
||||
|
||||
@@ -11,13 +11,13 @@
|
||||
|
||||
| Area | Status | Completion | Evidence |
|
||||
|------|--------|------------|----------|
|
||||
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_GDRIVE_TOKEN_WARN_DR_ESCROW_BLOCKED | 97% | 2026-06-25 09:37 live cold-start returned `PASS=87 WARN=1 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale and Google Drive token ownership/writeback metadata is not confirmed. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOOI API health is healthy/prod/mock=false, MOMO service health is healthy on `V10.657`, 110 / 188 runtime and backup checks are green。MOMO Gitea `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`; `cd.yaml #910` succeeded and deployed a fail-closed Drive auth/API boundary into 188 host source and `momo-scheduler` container source. Remaining hard service blocker is still MOMO business data freshness: `MOMO_DAILY_FRESHNESS 8|2026-06-17`; DB current-month parity remains `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`; latest valid job `56` is still completed with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Warning evidence: metadata-only check shows `/home/ollama/momo-pro/config/google_token.json` missing on host and `config/google_token.json` missing inside `momo-scheduler`, while scheduler runs as UID/GID `100000:100000`; no token content was read. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
|
||||
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_DATA_STALE_GDRIVE_TOKEN_WARN_DR_ESCROW_BLOCKED | 97% | 2026-06-25 10:23 live cold-start returned `PASS=87 WARN=1 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale and Google Drive token ownership/writeback metadata is not confirmed. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, AWOOOI API health is healthy/prod/mock=false, MOMO service health is healthy on `V10.657`, 110 / 188 runtime and backup checks are green。MOMO Gitea `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`; `cd.yaml #910` succeeded and deployed a fail-closed Drive auth/API boundary into 188 host source and `momo-scheduler` container source. The 10:04 scheduler cycle proved the live behavior: Google Drive auth failure is now logged as `❌ 自動匯入失敗` and sends Telegram failure notification, not no-file success. Remaining hard service blocker is still MOMO business data freshness: `MOMO_DAILY_FRESHNESS 8|2026-06-17`; DB current-month parity remains `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`; latest valid job `56` is still completed with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Warning evidence: metadata-only check shows `/home/ollama/momo-pro/config/google_token.json` missing on host and `config/google_token.json` missing inside `momo-scheduler`, while scheduler runs as UID/GID `100000:100000`; no token content was read. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
|
||||
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-25 09:05 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, VIP `192.168.0.125` is present, node filesystem / disk-pressure / readonly events are `0`, and latest `km-vectorize-29705460-55rgs` completed. |
|
||||
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-25 09:05 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`, last aggregate `2026-06-25 02:35:09`。DR remains blocked on real non-secret credential escrow evidence IDs. |
|
||||
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS_WITH_GDRIVE_TOKEN_WARN | 97% | Public route/TLS, API/Web route, MOMO health `V10.657`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. MOMO latest business date remains `2026-06-17`; stale age is `8` days as of 09:37. Latest valid job `56` already imported `即時業績_當日.xlsx` with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Google Drive token metadata is still a WARN because host and container token paths are missing; this requires owner-gated metadata repair/evidence and must not be solved by reading token contents. |
|
||||
| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE_V142_REPO_ONLY | 100% | Workplan, SOP v1.46, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with Google Drive token metadata WARN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
|
||||
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS_WITH_GDRIVE_TOKEN_WARN | 98% | Public route/TLS, API/Web route, MOMO health `V10.657`, MOMO main / CD `#904` monthly-sync failure boundary, MOMO main / CD `#910` Drive-auth fail-closed boundary, 10:04 live scheduler fail-closed proof, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. MOMO latest business date remains `2026-06-17`; stale age is `8` days as of 10:23. Latest valid job `56` already imported `即時業績_當日.xlsx` with `sync_success=true` and bounds `2026-06-01..2026-06-17`. Google Drive token metadata is still a WARN because host and container token paths are missing; this requires owner-gated metadata repair/evidence and must not be solved by reading token contents. |
|
||||
| P3 docs / automation contracts | DONE_WITH_MOMO_FAIL_CLOSED_LIVE_PROOF | 100% | Workplan, SOP v1.47, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence decision gate with scheduler stats / import_config / job 56 evidence, repo-side cold-start v1.42 source absence classifier, live-sync parity gate, MOMO import-boundary production deploy, MOMO Drive-auth fail-closed production deploy, 10:04 scheduler fail-closed live proof, MacBook Pro Codex safe artifact sync readback, and 2026-06-25 live refresh with Google Drive token metadata WARN are updated. 2026-06-24 23:15 read-only verify still shows repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; live 110 script sync of the v1.42 classifier is not claimed until separately approved and recorded. |
|
||||
|
||||
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-25 09:37, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, and MOMO service health is `V10.657`, but the latest repo-side live read-only cold-start scorecard remains `PASS=87 WARN=1 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and Google Drive token metadata is missing / writeback not confirmed. The hard blocker remains `188 momo source file absent while daily sales data stale`; the token state is a separate WARN and not a reason to read token contents. MOMO Drive auth/API failure is no longer allowed to be recorded as a no-file success after CD `#910`, but this code fix does not create new business data. This is repo-side source-of-truth evidence and not yet a claim that the 110 live monitor script was deployed. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
|
||||
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-25 10:23, routes/hosts/K3s/backups/exporters/monitoring surfaces are available, AWOOOI API is healthy, and MOMO service health is `V10.657`, but the latest repo-side live read-only cold-start scorecard remains `PASS=87 WARN=1 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and Google Drive token metadata is missing / writeback not confirmed. The hard blocker is now `188 momo daily sales data stale beyond 3 days`; the token state is a separate WARN and not a reason to read token contents. MOMO Drive auth/API failure is no longer allowed to be recorded as a no-file success after CD `#910`; the 10:04 scheduler run proved it now fails closed and sends failure notification. This code fix does not create new business data. This is repo-side source-of-truth evidence and not yet a claim that the 110 live monitor script was deployed. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
|
||||
|
||||
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user