wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

docs(ops): record 2104 recovery and momo v10.651 baseline [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-06-24 21:12:30 +08:00
parent b1858e7dcd
commit 9dbd31d945
4 changed files with 66 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
docs/LOGBOOK.md
View File

@@ -1,3 +1,27 @@
## 2026-06-24｜21:04 recovery readback 與 MOMO V10.651 雙機基準收斂
**背景**：前一輪 MOMO workspace readback 指到 `V10.646`，但 21:04 live health 已回 `V10.651`。因此本輪重新比對 Gitea `wooo/ewoooc` `main`、正式站 `/health`、Mac Mini / MacBook Pro Codex workspace 與 full-stack cold-start，避免「網站可用」和「版本 / 資料最新」互相混淆。
**Read-only service evidence**：
- Public route batch：`awoooi`、`vibework`、`awooogo`、`mo /health`、`stock`、`bitan`、`gitea`、`harbor`、`sentry`、`signoz`、`langfuse` 全部 `200`；`registry /v2/` 正常回 `401`。
- AWOOOI API health：`healthy / prod / mock_mode=false`，PostgreSQL、Redis、OpenClaw、SignOz、Ollama route providers 全部 `up`。
- Backup status no-notify：110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`offsite_fresh=1`，`rclone_gdrive_fresh=1`，`escrow_missing=5`，last backup-all `2026-06-24 02:28:39`。
- Full-stack cold-start：`PASS=86 WARN=0 BLOCKED=1`；110 / 120 / 121 / 188 network、188 runtime / exporters、110 registry / observability、K3s `mon` / `mon1` Ready、public routes/TLS、backup textfiles、110 systemd failed units、K8s active failed Jobs 均通過；唯一 blocker 仍是 `MOMO_DAILY_FRESHNESS 7|2026-06-17`。
- Stock ingestion：service route `200`，價格 / 三大法人 / market index 覆蓋 `2026-06-24`；`core.margin_short_daily` 仍因 `official_margin_short_daily` TWSE / TPEx `official_pending` 而 `blocked`，不是 route failure。
**MOMO release / workstation evidence**：
- Live `https://mo.wooo.work/health`：`{"database":"postgresql","status":"healthy","version":"V10.651"}`。
- Gitea `wooo/ewoooc` `main=65aa23800c7d0398f869225315bf96cb3d783c08`，`config.py` `SYSTEM_VERSION = "V10.651"`。
- `codex/momo-current-main-dev-base-20260624` 已快轉並推到 `65aa23800c7d0398f869225315bf96cb3d783c08`。
- Mac Mini `/Users/ogt/codex-workspaces/momo-pro-dev` 與 MacBook Pro `/Users/ooo/codex-workspaces/momo-pro-dev` 都在 branch `codex/momo-current-main-dev-base-20260624`，commit `65aa23800c7d0398f869225315bf96cb3d783c08`，dirty `0`，`SYSTEM_VERSION = "V10.651"`。
- `~/.codex/CODEX-START-HERE.md` 與 `~/.codex/codex-workstation-sync-dashboard.snapshot.json` 已同步到 MacBook Pro；local / remote SHA-256 一致，dashboard JSON parse 通過；未同步 auth、SQLite、sessions、raw conversations、`.env`、runtime volumes、raw `.git`。
**判定**：
- 可宣稱：核心網站 / K3s / backup / exporter / public route 已恢復；MOMO release 版本已對齊 Gitea main `V10.651`；Mac Mini / MacBook Pro MOMO Codex workspace 已同版。
- 不可宣稱：full-stack green、MOMO data current、DR complete。MOMO 業務資料仍停在 `2026-06-17`，credential escrow evidence 仍缺 `5`。
**邊界**：本輪沒有 Wazuh / SOC UI/API 修改，沒有 SSH 寫主機，沒有 Docker / Nginx / firewall / K8s / ArgoCD runtime 寫入，沒有 active scan，沒有讀 secret，沒有使用或保存聊天中的密碼。
## 2026-06-24｜MOMO V10.646 source-file absence 與雙機 Codex 基準收斂
**背景**：重啟恢復後，MOMO 的服務健康、程式版本、資料新鮮度與 MacBook / Mac Mini 開發基準不能混在一起判斷。`https://mo.wooo.work/health` 回 healthy 不代表業務資料已到今天；反過來，資料 stale 也不代表正式站仍跑舊版。

24
docs/runbooks/BACKUP-STATUS.md
View File

@@ -10,6 +10,30 @@
> 2026-06-13 Codex escrow refresh: 13:10 live report confirms offsite/rclone/script readiness is green and only five non-secret credential escrow evidence markers remain missing.
> 2026-06-18 Codex cold-start refresh: full-stack service readiness is green after stale failed Job classification; backup core remains green; DR still blocked only by five credential escrow evidence markers.
> 2026-06-24 Codex Velero/exporter refresh: 188 MinIO / Velero backup freshness, 188 PostgreSQL / Redis exporters, 188 node-exporter, and 110 disk pressure are recovered; DR still blocked only by five credential escrow evidence markers and service full-green is blocked by MOMO data freshness.
> 2026-06-24 21:04 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; full-stack remains blocked only by MOMO data freshness.
---
## 2026-06-24 21:04 Backup / Offsite / Escrow Live Status
Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110.
- 110 backup health: `13/13 fresh failed=0`。
- 188 backup health: `2/2 fresh failed=0`。
- Integrity / configured blockers: `core_blockers=0`、`configured_missing_110=0`、`configured_missing_188=0`、`script_missing_110=0`、`script_missing_188=0`。
- Offsite / GDrive freshness: `offsite_configured=1`、`offsite_fresh=1`、`rclone_gdrive_configured=1`、`rclone_gdrive_fresh=1`。
- Last aggregate backup: `2026-06-24 02:28:39`。
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
- Full-stack service release blocker remains separate: cold-start `PASS=86 WARN=0 BLOCKED=1`，原因是 `MOMO_DAILY_FRESHNESS 7|2026-06-17`；這不是 backup freshness failure。
| Gate | Status | Evidence |
|------|--------|----------|
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0. |
| 188 backup freshness | VERIFIED | 2/2 fresh, failed count 0. |
| Offsite / GDrive freshness | VERIFIED | `offsite_fresh=1`, `rclone_gdrive_fresh=1`. |
| Backup core blockers | GREEN | `core_blockers=0`. |
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
| Service full green | NO-GO | Blocked by MOMO data freshness, not by backup. |
---

26
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.35
> Version: v1.36
> Last updated: 2026-06-24 Asia/Taipei
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
@@ -10,29 +10,29 @@
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
2026-06-24 20:42 MOMO source-file and Codex dual-workstation readback supersede the earlier 20:17 wording where it discusses MOMO version / MacBook workspace state. The service and data readiness gates below are refreshed by the 20:17 live cold-start scorecard plus 20:42 source/workstation evidence:
2026-06-24 21:04 MOMO V10.651 and Codex dual-workstation readback supersede the earlier 20:42 wording where it discusses MOMO version / MacBook workspace state. The service and data readiness gates below are refreshed by the 21:04 live cold-start scorecard plus 21:06 source/workstation evidence:
```text
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
Live cold-start read-only check: PASS=86 WARN=0 BLOCKED=1, Result=BLOCKED.
Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
Runtime release state: API/Web/Worker are ready; latest deployment marker 622bc372 points runtime image to 2ec7f6f4 and production API health returns healthy. CD #3294 still has a historical Failure record because post-deploy monitoring coverage saw 188 nginx-exporter down before the exporter restore.
MOMO release state: mo.wooo.work health is healthy on version V10.646, matching Gitea main commit 7cfca9375445ea03d6f5d10512d0276a20914d25. Mac Mini and MacBook Pro controlled Codex workspaces are both on branch codex/momo-current-main-dev-base-20260624 at the same commit with dirty=0.
MOMO release state: mo.wooo.work health is healthy on version V10.651, matching Gitea main commit 65aa23800c7d0398f869225315bf96cb3d783c08. Mac Mini and MacBook Pro controlled Codex workspaces are both on branch codex/momo-current-main-dev-base-20260624 at commit 65aa23800c7d0398f869225315bf96cb3d783c08 with dirty=0.
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly match, but both stop at 2026-06-17. MOMO_DAILY_FRESHNESS is 7 days, which is a hard blocker because business data is not current.
Google Drive / source-file state: momo scheduler token ownership is fixed for Docker userns, container-side Drive listing works, but folder 當日業績匯入 currently has no matching 即時業績_當日 Excel source file. Archive latest matching file is 2026-06-18T01:30:39Z and was already imported by job 56. Mac Mini and MacBook candidate spreadsheets were also read-only inspected: the local current daily candidate only contains 2025-07-01..2025-07-02, the iCloud full-month candidate only contains 2025-06-01..2025-06-30, and MacBook candidates are either header-only or the same 2025-07-01..2025-07-02 dataset. These are not legitimate newer sources.
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, last aggregate is 2026-06-24 02:28:39, 188 MinIO is healthy, Velero BackupStorageLocation default is Available, one-off backup reboot-recovery-202606240456 completed, backup-health textfile reports Velero freshness green, PostgreSQL / Redis exporters are green, 188 nginx-exporter is restored with nginx_up=1, monitoring coverage is 14/14 jobs UP, and VeleroBackupNotRun / PostgreSQLDown / RedisDown / disk-pressure / nginx-exporter target-down evidence is resolved.
Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces are recovered; MOMO code release is current with Gitea main V10.646; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence.
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces are recovered; MOMO code release is current with Gitea main V10.651; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence.
Forbidden declaration: full-stack green, MOMO data current, DR complete, or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged.
```
2026-06-24 20:42 Codex workstation continuity readback:
2026-06-24 21:06 Codex workstation continuity readback:
```text
MacBook Pro 192.168.0.111 can authenticate to Gitea over SSH with its own public key named MacBook Pro Codex 20260624.
MOMO Pro Mac Mini workspace is /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 7cfca9375445ea03d6f5d10512d0276a20914d25, SYSTEM_VERSION V10.646, dirty=0.
MOMO Pro MacBook workspace is /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 7cfca9375445ea03d6f5d10512d0276a20914d25, SYSTEM_VERSION V10.646, dirty=0.
MOMO Pro Mac Mini workspace is /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 65aa23800c7d0398f869225315bf96cb3d783c08, SYSTEM_VERSION V10.651, dirty=0.
MOMO Pro MacBook workspace is /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 65aa23800c7d0398f869225315bf96cb3d783c08, SYSTEM_VERSION V10.651, dirty=0.
AwoooGo MacBook dev workspace remains ready at /Users/ooo/codex-workspaces/awooogo-dev, branch dev, upstream gitea/dev, commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty=0.
Safe handoff artifacts still match by readback after Start Here / workstation dashboard refresh; current SHA-256 values are tracked in the workstation dashboard artifact and local `sha256sum` readback rather than hard-coded here. Raw Codex App DB, auth, sessions, raw conversations, .env, runtime volumes, raw .git directories, passwords, tokens, and Mac Mini private keys were not copied.
This improves workstation continuity after host reboot / operator relocation, but does not change service cold-start status: full-stack green remains blocked by MOMO data freshness and DR remains blocked by credential escrow evidence.
@@ -214,9 +214,9 @@ DR_COMPLETE = no, because credential escrow evidence is incomplete
```text
110 / 120 / 121 / 188 HOST_READY = yes
Core public services SERVICE_READY = yes
MOMO_RELEASE_CURRENT = yes, because mo.wooo.work health is V10.646 and matches Gitea main commit 7cfca9375445ea03d6f5d10512d0276a20914d25
MOMO_RELEASE_CURRENT = yes, because mo.wooo.work health is V10.651 and matches Gitea main commit 65aa23800c7d0398f869225315bf96cb3d783c08
MOMO_DB_PARITY = yes
MOMO_DATA_FRESH = no, because latest daily_sales_snapshot date is 2026-06-17 and stale age is 7 days as of 2026-06-24 20:42
MOMO_DATA_FRESH = no, because latest daily_sales_snapshot date is 2026-06-17 and stale age is 7 days as of 2026-06-24 21:04
MOMO_SOURCE_AVAILABLE = no, because Drive intake has no newer 即時業績_當日 source and Mac Mini / MacBook candidate files only contain old or header-only data
FULL_STACK_GREEN = no, because cold-start scorecard is PASS=86 WARN=0 BLOCKED=1
DR_COMPLETE = no, because credential escrow evidence is incomplete
@@ -228,15 +228,15 @@ DR_COMPLETE = no, because credential escrow evidence is incomplete
重啟後若需要從 Mac Mini / MacBook Pro 繼續 Codex 開發，必須另外確認 Codex safe handoff artifacts，不得把服務恢復與 Codex raw 對話同步混為一談。
2026-06-24 20:42 Asia/Taipei readback：
2026-06-24 21:06 Asia/Taipei readback：
```text
MacBook Pro 192.168.0.111 SSH = OK
Safe artifacts synced = Start Here and workstation dashboard readback matched; current SHA-256 values are tracked in the workstation dashboard artifact and local sha256sum readback
Start Here readback = registry_ready 3, registry_blocked 8, latest_dev_on_gitea 3, production_on_gitea 8, raw_history_sync False
Workstation dashboard readback = artifact_sync_synced 2, artifact_sync_blocked 0, MOMO current main baseline ready 2
MOMO Pro Mac Mini workspace = /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 7cfca9375445ea03d6f5d10512d0276a20914d25, SYSTEM_VERSION V10.646, dirty 0
MOMO Pro MacBook workspace = /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 7cfca9375445ea03d6f5d10512d0276a20914d25, SYSTEM_VERSION V10.646, dirty 0
MOMO Pro Mac Mini workspace = /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 65aa23800c7d0398f869225315bf96cb3d783c08, SYSTEM_VERSION V10.651, dirty 0
MOMO Pro MacBook workspace = /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 65aa23800c7d0398f869225315bf96cb3d783c08, SYSTEM_VERSION V10.651, dirty 0
AwoooGo MacBook workspace = ready on dev commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty 0
```
@@ -252,7 +252,7 @@ MOMO Pro 可以在 Mac Mini / MacBook Pro 從 Gitea current-main Codex baseline
```text
raw Codex / ChatGPT 歷史聊天已同步。
所有產品都能雙機同步開發。
把 MOMO Pro 程式版本 V10.646 當成 MOMO 業務資料已更新。
把 MOMO Pro 程式版本 V10.651 當成 MOMO 業務資料已更新。
2026FIFA / Agent Bounty owner preflight 已通過。
```

10
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -11,13 +11,13 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED | 98% | 2026-06-24 20:17 live cold-start returned `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, 110 / 188 runtime and backup checks are green。188 `node-exporter`、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO / Velero BSL are restored; monitoring coverage is now `14/14 UP`; 110 disk pressure cleared。Remaining service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive listing works from the scheduler container, but `當日業績匯入` has no newer `即時業績_當日` Excel source file. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
| Overall recovery readiness | SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED | 98% | 2026-06-24 21:04 live cold-start returned `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED` because MOMO business data freshness remains stale. 110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, public routes/TLS are green, 110 / 188 runtime and backup checks are green。188 `node-exporter`、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO / Velero BSL are restored; monitoring coverage is now `14/14 UP`; 110 disk pressure cleared。Remaining service blocker is MOMO business data freshness: `MOMO_DAILY_FRESHNESS 7|2026-06-17`; Drive listing works from the scheduler container, but `當日業績匯入` has no newer `即時業績_當日` Excel source file. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-24 11:20 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`。188 `node-exporter` textfile scrape、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO endpoint、Velero BSL and latest completed backup freshness are restored; monitoring coverage is `14/14 UP`; `BackupHealthMonitorMissing188`、`PostgreSQLDown`、`RedisDown`、`VeleroBackupNotRun` and 110 disk-pressure alerts resolved. DR remains blocked on real non-secret credential escrow evidence IDs. |
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 96% | Public route/TLS, API/Web route, momo health `V10.646` matching Gitea main `7cfca9375445ea03d6f5d10512d0276a20914d25`, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. However MOMO latest business date is `2026-06-17`; stale age is `7` days as of 20:42. Drive pending folder has `0` matching files in repeated scheduler checks; Mac Mini / MacBook candidate files are old or header-only, so there is no safe newer source to import. |
| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.35, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MOMO V10.646 / Gitea main / dual-workstation Codex baseline readback, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 97% | 2026-06-24 21:04 backup / alert readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`。188 `node-exporter` textfile scrape、PostgreSQL exporter、Redis exporter、`nginx-exporter`、MinIO endpoint、Velero BSL and latest completed backup freshness are restored; monitoring coverage is `14/14 UP`; `BackupHealthMonitorMissing188`、`PostgreSQLDown`、`RedisDown`、`VeleroBackupNotRun` and 110 disk-pressure alerts resolved. DR remains blocked on real non-secret credential escrow evidence IDs. |
| P2 service / data truth | BLOCKED_MOMO_DATA_FRESHNESS | 96% | Public route/TLS, API/Web route, momo health `V10.651` matching Gitea main `65aa23800c7d0398f869225315bf96cb3d783c08`, current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness/storage conditions, VIP, and 110 / 188 runtime health are green. However MOMO latest business date is `2026-06-17`; stale age is `7` days as of 21:04. Drive pending folder has `0` matching files in repeated scheduler checks; Mac Mini / MacBook candidate files are old or header-only, so there is no safe newer source to import. |
| P3 docs / automation contracts | DONE_WITH_MOMO_SOURCE_ABSENCE_GATE | 100% | Workplan, SOP v1.36, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat Telegram suppression, MOMO scheduler / current-month detector fix, 188 node-exporter restore helper, 188 DB/Redis exporter restore helper, 188 MinIO/Velero restore helper, 188 nginx-exporter restore helper, 110 Docker disk pressure cleanup boundary, MOMO Google Drive token userns readback, MOMO daily freshness blocker, MOMO Pro false-noise health monitor source-of-truth, docker-health direct Telegram fallback cooldown, Bitan public-content same-fingerprint cooldown, notification-noise readback, MOMO source-file absence GO/NO-GO gate, MOMO V10.651 / Gitea main / dual-workstation Codex baseline readback, MacBook Pro Codex safe artifact sync readback, and MacBook Pro AwoooGo Gitea SSH / dev workspace readback are updated. Latest deploy marker `622bc372` points runtime image to `2ec7f6f4`; CD `#3294` retains a historical Failure because post-deploy monitoring coverage saw 188 `nginx-exporter` down before recovery, while manual coverage now passes `14/14 UP`. |
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-24 20:17, routes/hosts/K3s/backups/exporters/Velero/monitoring coverage are available, but the latest cold-start scorecard remains `PASS=86 WARN=0 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and no newer legitimate source file is available. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
Full cold-start service readiness may not be declared green for the latest verified evidence set. As of 2026-06-24 21:04, routes/hosts/K3s/backups/exporters/Velero/monitoring coverage are available, but the latest cold-start scorecard remains `PASS=86 WARN=0 BLOCKED=1` because MOMO business data freshness is stale beyond 3 days and no newer legitimate source file is available. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.