wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

docs(ops): record momo drive auth recovery readback [skip ci]

Browse Source
This commit is contained in:
ogt
2026-06-25 09:41:42 +08:00
parent eb7d92f110
commit 63f62ae49e
4 changed files with 36 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
docs/runbooks/BACKUP-STATUS.md
View File

@@ -16,12 +16,13 @@
> 2026-06-24 23:15 Codex live-sync gate readback: read-only deploy parity check correctly blocks because repo cold-start hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`; installer remains a live write requiring explicit approval.
> 2026-06-24 23:33 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; live cold-start still blocks only on MOMO source absence / data freshness, not backup.
> 2026-06-25 09:05 Codex backup readback: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `escrow_missing=5`; live cold-start is `PASS=87 WARN=1 BLOCKED=1` because MOMO business data is stale and MOMO Google Drive token metadata is missing.
> 2026-06-25 09:37 Codex MOMO deploy readback: MOMO `main` is `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`, Gitea Actions `cd.yaml #910` succeeded, and 188 host / `momo-scheduler` source now fail closed on Google Drive auth/API failure. Backup/offsite remains green; full-stack still blocks on MOMO data freshness and `escrow_missing=5`.
---
## 2026-06-25 09:05 Backup / Offsite / Escrow Live Status
## 2026-06-25 09:37 Backup / Offsite / Escrow Live Status
Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 09:05 Asia/Taipei.
Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` from 110 at 09:33 Asia/Taipei; cold-start rerun at 09:37.
- 110 backup health: `13/13 fresh failed=0`。
- 188 backup health: `2/2 fresh failed=0`。
@@ -30,6 +31,7 @@ Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` f
- Last aggregate backup: `2026-06-25 02:35:09`。
- DR blocker remains: `escrow_missing=5`，不得偽造 evidence marker，也不得貼 secret value / hash / partial token。
- Full-stack service release blocker remains separate: cold-start `PASS=87 WARN=1 BLOCKED=1`，原因是 MOMO business data freshness stale (`MOMO_DAILY_FRESHNESS 8|2026-06-17`) plus Google Drive token metadata missing / writeback not confirmed。這不是 backup freshness failure。
- MOMO code boundary now covers both failure modes: `cd.yaml #904` makes monthly sync failure fail the import job and prevents Drive file movement; `cd.yaml #910` makes Drive auth/API failure return `success=false` instead of a no-file success.
| Gate | Status | Evidence |
|------|--------|----------|
@@ -39,6 +41,7 @@ Read-only command: `/backup/scripts/backup-status.sh --no-notify --no-refresh` f
| Backup core blockers | GREEN | `core_blockers=0`. |
| Credential escrow | BLOCKED | `escrow_missing=5`; only real non-secret owner evidence may close this. |
| MOMO Drive token metadata | WARN | Host and container token metadata paths are missing; no token content was read. |
| MOMO Drive auth false-green | FIX_DEPLOYED | Gitea Actions `cd.yaml #910` success; 188 host and scheduler container source include fail-closed marker. |
| Service full green | NO-GO | Blocked by MOMO source absence / data freshness; token metadata warning also requires owner-gated evidence. |
---

14
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.45
> Version: v1.46
> Last updated: 2026-06-25 Asia/Taipei
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
@@ -10,22 +10,22 @@
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
2026-06-25 09:05 live read-only refresh supersedes the 2026-06-24 23:33 scorecard wording. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, and backup/offsite are available, while full-stack release remains blocked by MOMO source absence / business data freshness and DR remains blocked by missing credential escrow evidence. A new WARN shows MOMO Google Drive token ownership/writeback is not confirmed because the token metadata path is missing from both host and container views; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
2026-06-25 09:37 live read-only refresh supersedes the 09:05 wording. It confirms the SOP is behaving correctly: hosts, routes, K3s, AWOOOI API health, MOMO service health, and backup/offsite are available, while full-stack release remains blocked by MOMO source absence / business data freshness and DR remains blocked by missing credential escrow evidence. MOMO Drive auth false-green is now fixed in production by Gitea Actions `cd.yaml #910` at commit `e137d7a5d02a7595a44c3f3cc1cf54b766424ee7`: auth/API failure now fails closed instead of being reported as an empty folder. The Google Drive token ownership/writeback WARN remains because token metadata is still missing; this must be handled as a separate evidence / owner gate and not by reading token contents. The 110 live script parity blocker from 23:15 still applies.
```text
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
Live cold-start read-only check: 2026-06-25 09:05 PASS=87 WARN=1 BLOCKED=1, Result=BLOCKED.
Live cold-start read-only check: 2026-06-25 09:37 PASS=87 WARN=1 BLOCKED=1, Result=BLOCKED.
Repo-side cold-start v1.42 live read-only run: New MOMO fields remain MOMO_SOURCE_EMPTY_EVIDENCE_LINES=21, MOMO_IMPORT_CONFIG=當日業績匯入|即時業績_當日, MOMO_LATEST_IMPORT_JOB=56|completed|即時業績_當日.xlsx|2026-06-18T11:41:00.853176|2026-06-18T11:42:02.309425|10936|10936|0. The only BLOCKED text is "188 momo source file absent while daily sales data stale". Live 110 script sync is not claimed until a separate approved deployment/sync happens.
110 live-sync parity: 2026-06-24 23:15 read-only `verify-cold-start-monitor-deploy.sh` correctly BLOCKED because repo script hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`. Do not use live 110 monitor output to prove v1.42 behavior until the approved live-sync gate in §13.3.1 passes.
Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
Runtime release state: API/Web/Worker are ready; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. `ollama_local` had a short cooldown in the direct health JSON and is not the current release blocker. 09:05 route smoke shows awoooi API=200, `/zh-TW/iwooos`=200, vibework=200, awoooogo=200, momo health=200, stock=200, bitan=200, gitea=200, harbor=200, sentry=200, signoz=200, langfuse=200, aiops=200, registry /v2=401; cold-start raw route gate still records expected redirect statuses such as awoooi web=307 and sentry=302.
MOMO release state: mo.wooo.work health is healthy on version V10.655. Gitea main fast-forwarded to 84035906aba0e5e190d031a13cfd9b47a8cd1f73 and Gitea Actions cd.yaml #904 completed Success for the import-boundary fix; later version readback confirms the service is current enough to separate release health from business-data freshness.
MOMO release state: mo.wooo.work health is healthy on version V10.657. Gitea main fast-forwarded to e137d7a5d02a7595a44c3f3cc1cf54b766424ee7 and Gitea Actions cd.yaml #910 completed Success for the Drive-auth fail-closed fix. 188 host source and `momo-scheduler` container source both contain the marker `Google Drive 連線或認證失敗，未能確認來源資料夾是否有新檔案`; scheduler / telegram-bot were restarted by CD and are healthy. Earlier cd.yaml #904 remains the monthly-sync failure boundary fix.
MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly still match, but both stop at 2026-06-17: `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`. `MOMO_DAILY_FRESHNESS 8|2026-06-17` is a hard blocker because business data is not current.
Google Drive / source-file state: 2026-06-25 09:05 cold-start reports `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`; direct metadata-only readback confirms host path `/home/ollama/momo-pro/config/google_token.json` is missing and container-side `config/google_token.json` is missing, while the scheduler process runs as UID/GID `100000:100000`. Do not read token content and do not recreate/chown token evidence without an explicit maintenance-window / owner approval. The data blocker remains source absence / stale business data; the token missing state is a separate WARN until owner-provided token/writeback evidence is restored.
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09. 09:05 backup-status --no-notify --no-refresh reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, integrity_stale=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5.
Google Drive / source-file state: 2026-06-25 09:37 cold-start reports `MOMO_GDRIVE_TOKEN_STAT missing scheduler_uid=100000`; direct metadata-only readback confirms host path `/home/ollama/momo-pro/config/google_token.json` is missing and container-side `config/google_token.json` is missing, while the scheduler process runs as UID/GID `100000:100000`. Do not read token content and do not recreate/chown token evidence without an explicit maintenance-window / owner approval. The data blocker remains source absence / stale business data; the token missing state is a separate WARN until owner-provided token/writeback evidence is restored. With cd.yaml #910 live, any future Drive auth/API failure must be treated as failed import evidence rather than a no-file success.
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09. 09:33 backup-status --no-notify --no-refresh reports 110 13/13 fresh failed=0, 188 2/2 fresh failed=0, core_blockers=0, integrity_stale=0, offsite_fresh=1, rclone_gdrive_fresh=1, escrow_missing=5.
Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces, AWOOOI API health, and MOMO service health are available for the latest read-only evidence set; MOMO production code includes the import-boundary fix; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence; MOMO Google Drive token/writeback evidence needs a separate non-secret owner-gated repair.
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces, AWOOOI API health, and MOMO service health are available for the latest read-only evidence set; MOMO production code includes the monthly-sync failure boundary and Drive-auth fail-closed fixes; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence; MOMO Google Drive token/writeback evidence needs a separate non-secret owner-gated repair.
Forbidden declaration: full-stack green, MOMO data current, DR complete, credential escrow complete, 110 live monitor synced, or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged.
```