wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

fix(ops): 區分 stale failed Job cold-start 判定 [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-06-18 13:53:55 +08:00
parent c7597df232
commit f89f59c647
6 changed files with 168 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
docs/LOGBOOK.md
View File

@@ -170,6 +170,54 @@
**邊界**：本輪只補 repo 文件、snapshot / guard、前台只讀可視化與 production smoke；未 SSH、未改 Nginx、未改 firewall、未 reload、未 kubectl / ArgoCD sync、未 Wazuh active response、未 Kali active scan、未 Kali `/execute`、未更新套件、未讀或保存 secret、未建立 SOAR case、未送 Telegram、未做 production runtime 寫入。
## 2026-06-18｜110 CPU 高負載只讀診斷與孤兒 Chrome 清理
**背景**：使用者詢問為何 110 主機 CPU / load 又持續滿載。為避免誤判成 AWOOOI cold-start、Harbor、K3s 或 Docker restart 問題，本輪只讀檢查 110 的 uptime、CPU pressure、memory / swap、vmstat、process table、Docker stats、Gitea runner activity 與 systemd failed units。
**診斷結果**：
- 13:48 110 load average 為 `14.03 / 11.15 / 8.50`，CPU cores 為 `12`；`systemctl is-system-running` 為 `running`，failed units `0`。
- 記憶體不是主要瓶頸：`Mem available` 約 `39-40Gi`；Swap 仍為 `7.8Gi / 7.8Gi`，但 vmstat 當下 `si=0`、`so=0`，沒有明顯即時 swap thrash。
- 第一主因是跨專案 `stockPlatform` headless Chrome / Playwright 類 smoke 留下孤兒程序，尤其 `/tmp/stockplatform-review-bulk-ux-ZaVuMP` 與 `/tmp/stockplatform-review-bulk-ux-vGdwKs` 各約 `120%` CPU，已跑超過 4 小時。
- 第二主因是同時有多個 Gitea Actions 任務在跑：`awoooi-cd-4501-1-api-tests`、`2026 World Cup Quant Platform`、`VibeWork CI`，其中 pytest / pnpm / pip / apt 會造成短期 CPU 尖峰。
- Docker / Sentry / Harbor / Gitea 是 110 的固定背景負載來源，但不是本次最高尖峰的根因。
**處置**：
- 只對父程序已是 PID 1、執行超過 2 小時、`user-data-dir` 為 `/tmp/stockplatform-review-bulk-ux-*` 的 orphaned Chrome process group 送出 `SIGTERM`。
- 清理目標 process groups：`1996741`、`2090510`、`2504537`、`2557310`、`3185294`。
- `REMAINING_AFTER_TERM=0`；未清理正在有 node parent 的 `stockplatform-product-ux-smoke.mjs`，未中止 AWOOOI CD / World Cup / VibeWork CI，未動 Docker / systemd / Nginx / firewall。
**後續讀回**：
- 13:50 load average 開始從尖峰回落，但仍為 `10.20 / 10.55 / 8.62`，因為 load average 是滑動平均，且 Gitea Actions 測試仍在跑。
- 13:50 即時 vmstat 顯示 CPU idle 約 `63-65%`，表示孤兒 Chrome 清掉後，主機已不再處於前一段 Chrome GPU process 多核心持續忙等狀態。
- Swap 仍滿，短期不建議在 CI / runner / Sentry 背景負載仍高時手動 `swapoff -a && swapon -a`；若要清 swap，應先等 Gitea Actions 降載並確認 available memory 穩定。
**邊界**：本輪未 restart Docker、未 stop container、未改 Nginx、未改 firewall、未 kill AWOOOI / Harbor / Sentry / Gitea 服務、未讀 secret、未部署、未 active scan。這是針對 orphaned browser smoke 的精準清理，不是主機重啟或服務恢復操作。
## 2026-06-18｜cold-start stale failed Job 分類修正與 live GREEN readback
**背景**：12:17 live readback 顯示服務可用但仍因保留舊 `km-vectorize-29689620` failed Job 被標成 `DEGRADED`。這和 `failedJobsHistoryLimit=3` 的 evidence-retention 目的衝突：我們需要保留舊失敗作為證據，但不能在同一 CronJob 後續已有官方成功 Job 後，讓 cold-start 永久保持 warning。
**完成內容**：
- `scripts/reboot-recovery/full-stack-cold-start-check.sh` 將 K8s Job 判定拆成 `FAILED_JOBS`、`STALE_FAILED_JOBS`、`ACTIVE_FAILED_JOBS`。
- 同一 CronJob 若 retained failed Job 之後已有更晚的 `Complete` Job，該 failed Job 只算 `STALE_FAILED_JOBS`，不再造成 cold-start warning。
- 若 failed Job 沒有後續成功，仍算 `ACTIVE_FAILED_JOBS`，cold-start 繼續 warning，避免 false-green。
- `scripts/reboot-recovery/reboot-recovery-readiness-audit.sh` 新增 `STALE_FAILED_JOBS` 與 `ACTIVE_FAILED_JOBS` 合約檢查，避免未來回退成「看到任何 failed Job 就永久 warning」或「完全忽略 failed Job」。
**驗證**：
- `bash -n scripts/reboot-recovery/full-stack-cold-start-check.sh scripts/reboot-recovery/reboot-recovery-readiness-audit.sh` 通過。
- `PATH=/tmp/awoooi-ansible-venv/bin:$PATH bash scripts/reboot-recovery/reboot-recovery-readiness-audit.sh --no-color` 回 `PASS=187 WARN=1 BLOCKED=0`，唯一 warning 仍是 repo-side audit 未帶 `--live`。
- `bash scripts/reboot-recovery/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` 回 `PASS=84 WARN=0 BLOCKED=0`，result `GREEN`。
- Live schedule evidence：`FAILED_JOBS 1`、`STALE_FAILED_JOBS 1`、`ACTIVE_FAILED_JOBS 0`、`BAD_PODS 0`。
- 110 / 120 / 121 / 188 ping 與 SSH port 全部 OK；120 / 121 K3s nodes Ready；`NODE_FS_ERROR_EVENTS 0`；AWOOOI API/Web routes OK；public routes / TLS green；momo current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`；110 backup health `13/13 fresh failed=0`；188 backup health `2/2 fresh failed=0`。
**完成度同步**：
- Full cold-start service readiness：`SERVICE_AVAILABLE_DEGRADED -> FULL_STACK_GREEN_FOR_SERVICE`。
- Reboot SOP / Plan B / automation contracts：`100%`。
- Overall recovery readiness：`99% SERVICE_GREEN_DR_ESCROW_BLOCKED`。
- DR complete：仍 `NO-GO`，credential escrow evidence markers 不可偽造，仍需真實非秘密 owner evidence IDs。
**邊界**：本輪未刪除 failed Job、未手動建立 Job、未 patch K8s、未 ArgoCD sync、未重啟服務、未改 Docker / Nginx / firewall、未讀 secret、未送 Telegram、未 active scan。這次修正只把 cold-start 判定從「保留舊 failed Job 就永遠 DEGRADED」改成「active failed Job 才警告；stale failed Job 保留為 evidence」。
## 2026-06-18｜重啟 live cold-start readback：服務可用但保留 stale failed Job warning
**背景**：重啟 SOP / Plan B / repo-side readiness blockers 已推上 `gitea/main=63d8361f` 後，為避免把 repo-side readiness 誤講成 live full green，本輪立即用只讀方式重跑 cold-start gate 並追蹤剛好同時發生的 AWOOOI rollout 自然收斂。

24
docs/runbooks/BACKUP-STATUS.md
View File

@@ -8,6 +8,30 @@
> 2026-06-13 Codex live refresh: backup core remains green; DR still blocked only by credential escrow evidence.
> 2026-06-13 Codex post-CD refresh: backup/offsite/alert contracts remain green after deploy marker `e4a349bc`; global SSH trust guardrail held; DR still blocked only by credential escrow evidence.
> 2026-06-13 Codex escrow refresh: 13:10 live report confirms offsite/rclone/script readiness is green and only five non-secret credential escrow evidence markers remain missing.
> 2026-06-18 Codex cold-start refresh: full-stack service readiness is green after stale failed Job classification; backup core remains green; DR still blocked only by five credential escrow evidence markers.
---
## 2026-06-18 Cold-Start / Backup Live Status
2026-06-18 13:43 refresh:
- `full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1`：`PASS=84 WARN=0 BLOCKED=0`，result `GREEN`。
- K8s schedule evidence：`FAILED_JOBS=1`、`STALE_FAILED_JOBS=1`、`ACTIVE_FAILED_JOBS=0`、`BAD_PODS=0`。Retained `km-vectorize` failed Job is historical evidence only; active failed Job count is zero.
- 110 backup health：`total=13 stale=0 missing_cron=0 missing_script=0 failed_count=0 config_failed=0 integrity_total=2 integrity_stale=0`。
- 188 backup health：`total=2 stale=0 missing_cron=0 missing_script=0`。
- Public routes / TLS, momo DB parity, 120 / 121 K3s readiness, AWOOOI API/Web route checks all passed in the same cold-start run.
- Credential escrow readback remains blocked: `/backup/scripts/offsite-escrow-evidence-report.sh --no-color` reports `SCRIPT_MISSING_COUNT=0`、`OFFSITE_CONFIGURED=1`、`RCLONE_CONFIGURED=1`、`READINESS_REQUIRE_CONFIGURED_BLOCKED=0`、`ESCROW_MISSING_COUNT=5`。
| Gate | Status | Evidence |
|------|--------|----------|
| Full cold-start service readiness | GREEN | `PASS=84 WARN=0 BLOCKED=0`; stale failed Job evidence is separated from active failed Job blockers. |
| 110 backup freshness | VERIFIED | 13/13 fresh, failed count 0, config failed 0, integrity stale 0. |
| 188 backup freshness | VERIFIED | 2/2 fresh, missing cron/script 0. |
| Credential escrow | BLOCKED | Five non-secret evidence markers still missing: `restic_repository_password`, `offsite_provider_credentials`, `break_glass_admin_credentials`, `dns_registrar_recovery`, `oauth_ai_provider_recovery`. |
| DR closeout | NO-GO | Must not be declared complete until real owner-provided non-secret evidence IDs are validated and markers are written. |
Current policy: service recovery and backup health can be green while DR is still blocked. Do not fake escrow markers, do not paste secrets into repo/chat, and do not silence escrow alerts.
---

57
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.24
> Version: v1.25
> Last updated: 2026-06-18 Asia/Taipei
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
@@ -22,26 +22,37 @@ Allowed declaration: services are available with one stale failed Job warning.
Forbidden declaration: full cold-start green, DR complete, or runtime/security acceptance.
```
| 項目 | 2026-06-14 18:15 Asia/Taipei live result | 判定 |
2026-06-18 13:43 live readback supersedes the stale-Job warning wording:
```text
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
Live cold-start read-only check: PASS=84 WARN=0 BLOCKED=0, Result=GREEN.
Service state: FULL_STACK_GREEN_FOR_SERVICE; 110/120/121/188 reachable, K3s mon/mon1 Ready, NODE_FS_ERROR_EVENTS=0, public routes/TLS green, 110/188 backup health fresh.
K8s Job classification: FAILED_JOBS=1, STALE_FAILED_JOBS=1, ACTIVE_FAILED_JOBS=0. The retained km-vectorize failure stays as evidence but no longer blocks service readiness after later official successful Jobs.
Allowed declaration: full cold-start service readiness is green for this evidence set.
Forbidden declaration: DR complete or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged.
```
| 項目 | 2026-06-18 13:43 Asia/Taipei live result | 判定 |
|------|-------------------------------------------|------|
| Overall recovery readiness | `97%` | `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
| Overall recovery readiness | `99%` | `SERVICE_GREEN_DR_ESCROW_BLOCKED` |
| P0 host / K3s recovery | `100%` | `DONE` |
| P1 backup / alert / escrow | `92%` | `BLOCKED_DR_ESCROW` |
| P2 service / data truth | `99%` | `VERIFIED_WORKLOAD_BALANCED_WITH_KM_WARN` |
| P3 docs / automation contracts | `100%` | `DONE_WITH_VALIDATION_GAP` |
| P2 service / data truth | `100%` | `VERIFIED_FULL_STACK_GREEN_FOR_SERVICE` |
| P3 docs / automation contracts | `100%` | `DONE_WITH_STALE_JOB_CLASSIFICATION` |
| 110 host runtime | `fwupd-refresh.timer` intentionally disabled/inactive after non-runtime firmware metadata refresh failed units were classified; `systemctl --failed` returns `0 loaded units listed`; rollback is `sudo systemctl enable --now fwupd-refresh.timer` | `GREEN_WITH_FWUPD_TIMER_DISABLED` |
| 120 reachability | ping OK, SSH OK, boot around `2026-06-14 02:23`, K3s active, node `mon Ready` | `GREEN` |
| 121 reachability | ping OK, SSH OK, failed units `0` | `GREEN` |
| 188 host runtime | production services green; host degraded only by `certbot.service` and `snap.certbot.renew.service` | `GREEN_WITH_CERTBOT_DEBT` |
| K3s node state | `mon Ready control-plane`, `mon1 Ready control-plane`; latest readback keeps API/Web split across 120 / 121; bad pods `0`; retained failed Job `km-vectorize-29689620` is the only WARN | `GREEN_WORKLOAD_BALANCED_WITH_KM_WARN` |
| K3s node state | `mon Ready control-plane`, `mon1 Ready control-plane`; bad pods `0`; `FAILED_JOBS=1`, `STALE_FAILED_JOBS=1`, `ACTIVE_FAILED_JOBS=0` | `GREEN_WITH_RETAINED_EVIDENCE` |
| 110 -> 120 / 188 SSH trust | 00:33 cold-start exposed stale `known_hosts`; backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; final repair backup `/home/wooo/.ssh/known_hosts.before-120-188-final-refresh.20260613-011949`; CD fix `80e6ec1a` moves deploy trust to `/home/wooo/.ssh/deploy_known_hosts`; 01:28 global `known_hosts` still contains 120 / 188 and was not clobbered by deploy marker `e4a349bc` | `GREEN_WITH_GUARDRAIL` |
| Backup status | 18:14 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22` | `GREEN_WITH_DR_ESCROW_WARNING` |
| Backup status | 13:43 status: 110 backup health `13/13 fresh failed=0`, 188 backup health `2/2 fresh failed=0`; escrow readback still shows `ESCROW_MISSING_COUNT=5` | `GREEN_WITH_DR_ESCROW_WARNING` |
| Offsite sync / verify | 01:28 textfile: `awoooi_backup_offsite_remote_verify_ok=1`, `full_verify_fresh=1`, all 13 repos have `snapshot_count=1` and `snapshot_latest_only=1`; latest scheduled verifier log is 2026-06-12 07:20 | `GREEN` |
| Backup / cold-start alerts | 01:27 live visibility check confirms Prometheus and Alertmanager expose the 5 required credential escrow gap alerts; Prometheus rules API has all five required alert names healthy; label contract check loads 24 baseline backup alert rules | `GREEN_WITH_EXPECTED_REDLIGHTS` |
| Cold-start scorecard | 18:15 高價值配置 Owner Packet 前台同步後只讀 scorecard：`PASS=82 WARN=1 BLOCKED=0`；remaining WARN 仍是 K8s failed Job `km-vectorize-29689620`。Public routes / TLS、momo DB parity、backup exporters、120/121 K3s 與 AWOOOI API/Web 皆通過。 | `DEGRADED_NO_BLOCKERS` |
| Cold-start scorecard | 13:43 stale Job classification 後只讀 scorecard：`PASS=84 WARN=0 BLOCKED=0`。Public routes / TLS、momo DB parity、backup exporters、120/121 K3s 與 AWOOOI API/Web 皆通過。 | `GREEN_FOR_SERVICE` |
| momo DB parity | `4571|4571|2026-06-01|2026-06-07|2026-06-01|2026-06-07` | `GREEN` |
| momo scheduler | container healthy; scorecard reads `SCHEDULER_RECENT_ACTIVITY 1009`; detector widened and deployed to 110 | `GREEN` |
| ArgoCD app health | Latest live app source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94` remains `Synced / Degraded` because official Job `km-vectorize-29689620` failed and CronJob health reports `CronJob has not completed its last execution successfully`. Latest repo docs head is `0a4766dd`, but runtime image readback is from deploy marker `16c6b983` / feature `e999c16b`. CronJob `lastScheduleTime=2026-06-13T19:00:00Z` and `lastSuccessfulTime=2026-06-04T11:00:37Z`; failed Job is retained, but its failed Pod/log are not currently available, so the next official 03:00 run remains the required read-only evidence gate. | `WAITING_OFFICIAL_KM_VECTORIZE_RUN` |
| ArgoCD app health | This 13:43 cold-start pass does not reassert ArgoCD app health directly. Service release uses K8s node / route / workload / CronJob active-failure evidence. Any ArgoCD release claim still requires a fresh `awoooi-prod` app readback. | `NOT_REASSERTED_BY_THIS_GATE` |
| Workload balancing | Live API/Web/Worker/CronJob image is `e999c16b3435f197b78fe2adfeec1c4faa6c4675`; API/Web pods remain split across `mon` / `mon1`, Worker single replica remains healthy on `mon` | `GREEN` |
| Credential escrow | 5 non-secret evidence markers missing | `BLOCKED` |
@@ -78,6 +89,17 @@ Do not delete the failed Job manually during routine SOP verification. Keep it a
Current allowed wording: "SOP / Plan B / automation contracts are complete; live services are available with one retained stale km-vectorize failed Job warning; hard blockers are zero; DR remains blocked by credential escrow evidence."
```
2026-06-18 13:43 live rule:
```text
GO for full cold-start service readiness for this evidence set: PASS=84 WARN=0 BLOCKED=0.
GO for controlled runner/CD release under the normal security gates; this is not a bypass for owner response, runtime writer, Telegram, Gateway, K8s, Docker, Nginx, firewall, or secret operations.
GO for retaining stale failed Job evidence: FAILED_JOBS=1 and STALE_FAILED_JOBS=1 are allowed when ACTIVE_FAILED_JOBS=0 and later official successful Jobs exist.
NO-GO for DR complete while credential escrow evidence markers remain missing: ESCROW_MISSING_COUNT=5.
NO-GO for deleting retained failed Jobs during routine verification. Cleanup requires an explicit maintenance window and owner acceptance.
Current allowed wording: "full-stack service recovery is green for the current evidence set; stale km-vectorize failure is retained as historical evidence, not an active blocker; DR complete remains blocked by credential escrow evidence."
```
After any future 120 recovery, rerun this exact chain from 110:
```bash
@@ -1521,6 +1543,23 @@ SOP update:
| Remaining warning | retained stale Job `km-vectorize-29689620` from 2026-06-14 03:00; later official Jobs `km-vectorize-29692500`, `29693940`, `29695380` are `Complete` |
| Declaration limit | 可宣稱 `SERVICE_AVAILABLE_DEGRADED`；不可宣稱 `FULL_STACK_GREEN`，因為 `WARN=1`；不可宣稱 `DR_COMPLETE`，credential escrow evidence still requires real non-secret owner evidence |
### 14.25 2026-06-18 stale failed Job classification and service-green readback
2026-06-18 13:43 的變更不是刪除 K8s Job，也不是手動建立 Job，而是修正 cold-start 判定邏輯：保留的歷史 failed Job 是 evidence；只有沒有後續官方成功 Job 的 failed Job 才是 active blocker。這讓 evidence retention 和 service readiness 不再互相衝突。
| 項目 | 2026-06-18 13:43 stale Job classification baseline |
|------|----------------------------------------------------|
| SOP version | `v1.25` |
| Script change | `full-stack-cold-start-check.sh` emits `FAILED_JOBS`, `STALE_FAILED_JOBS`, and `ACTIVE_FAILED_JOBS` |
| Active blocker rule | `ACTIVE_FAILED_JOBS > 0` causes warning; `STALE_FAILED_JOBS > 0` is retained evidence and does not warn by itself |
| Readiness audit contract | `reboot-recovery-readiness-audit.sh` requires both stale and active failed Job counters |
| Repo-side validation | `bash -n` passed; readiness audit returned `PASS=187 WARN=1 BLOCKED=0` with only the expected non-live warning |
| Live cold-start readback | `PASS=84 WARN=0 BLOCKED=0`, result `GREEN` |
| K8s Job evidence | `FAILED_JOBS=1`, `STALE_FAILED_JOBS=1`, `ACTIVE_FAILED_JOBS=0`, `BAD_PODS=0` |
| Backup / DR evidence | 110 backup health `13/13 fresh failed=0`; 188 backup health `2/2 fresh failed=0`; escrow readback still `ESCROW_MISSING_COUNT=5` |
| Declaration limit | 可宣稱 `FULL_STACK_GREEN_FOR_SERVICE`；不可宣稱 `DR_COMPLETE`、`credential escrow complete` 或任何 runtime/security acceptance |
| SOP change | v1.25 defines retained failed Job evidence vs active failed Job blocker; future reboot comparison must record all three counters |
### 14.22 重啟後時間軸驗證
每次重啟後照時間軸推進，不要等到最後才一次判定。

20
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -11,13 +11,13 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | SERVICE_AVAILABLE_DEGRADED_STALE_KM_JOB_DR_ESCROW_BLOCKED | 98% | 2026-06-18 12:13 live cold-start read-only gate returned `PASS=83 WARN=1 BLOCKED=0`, result `DEGRADED`。110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, `NODE_FS_ERROR_EVENTS=0`, public routes/TLS are green, 110 backup health is `13/13 fresh failed=0`, 188 backup health is `2/2 fresh failed=0`, and final rollout readback at 12:17 shows API `2/2`, Web `2/2`, Worker `1/1`, Canary `1/1`, API health `200 healthy`。Only warning is retained stale Job `km-vectorize-29689620` from 2026-06-14 03:00; later official `km-vectorize` Jobs `29692500` / `29693940` / `29695380` are `Complete`。DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
| Overall recovery readiness | SERVICE_GREEN_DR_ESCROW_BLOCKED | 99% | 2026-06-18 13:43 live cold-start read-only gate returned `PASS=84 WARN=0 BLOCKED=0`, result `GREEN`。110 / 120 / 121 / 188 ping and SSH port are OK, K3s `mon` / `mon1` are Ready, `NODE_FS_ERROR_EVENTS=0`, public routes/TLS are green, 110 backup health is `13/13 fresh failed=0`, 188 backup health is `2/2 fresh failed=0`。K8s schedule readback now distinguishes `FAILED_JOBS=1` / `STALE_FAILED_JOBS=1` / `ACTIVE_FAILED_JOBS=0`; the retained `km-vectorize-29689620` failed Job is historical evidence, not an active service blocker. DR remains blocked because credential escrow evidence markers are still missing and must not be forged. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-15 03:11 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-15 02:40:13`. Offsite / escrow report shows `SCRIPT_MISSING_COUNT=0`, `OFFSITE_CONFIGURED=1`, `RCLONE_CONFIGURED=1`, `ESCROW_MISSING_COUNT=5`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. |
| P2 service / data truth | VERIFIED_SERVICE_AVAILABLE_WITH_STALE_JOB_WARN | 99% | 2026-06-18 12:13 cold-start verifies public route/TLS, API/Web route, momo health and current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness, VIP, and 110 / 188 runtime health. 12:17 post-rollout readback confirms AWOOOI deployments ready and API health `200 healthy`. The only service warning is old retained `km-vectorize-29689620` failed Job evidence; no hard blocker remains. |
| P3 docs / automation contracts | REPO_SIDE_READY_LIVE_CHECK_RUN_WITH_WARN | 100% | Workplan, SOP v1.24, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, and 2026-06-18 live readback are updated. Repo-side `reboot-recovery-readiness-audit.sh --no-color` returned `PASS=185 WARN=1 BLOCKED=0`; live cold-start returned `PASS=83 WARN=1 BLOCKED=0`. |
| P2 service / data truth | VERIFIED_FULL_STACK_GREEN_FOR_SERVICE | 100% | 2026-06-18 13:43 cold-start verifies public route/TLS, API/Web route, momo health and current-month parity `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17`, backup exporters, schedules, K3s node readiness, VIP, and 110 / 188 runtime health. K8s active failed Job count is `0`, bad pods are `0`, and cold-start returns `PASS=84 WARN=0 BLOCKED=0`. |
| P3 docs / automation contracts | DONE_WITH_STALE_JOB_CLASSIFICATION | 100% | Workplan, SOP v1.25, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, and 2026-06-18 live readback are updated. Repo-side `reboot-recovery-readiness-audit.sh --no-color` returned `PASS=187 WARN=1 BLOCKED=0`; live cold-start returned `PASS=84 WARN=0 BLOCKED=0`. |
Full cold-start may be declared green only for the latest verified evidence set. As of 2026-06-18 12:17, services are available and hard blockers are zero, but the latest scorecard is still `DEGRADED` by one retained stale `km-vectorize` failed Job warning. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
Full cold-start service readiness may be declared green for the latest verified evidence set. As of 2026-06-18 13:43, services are green with `WARN=0` and `BLOCKED=0`; the retained stale `km-vectorize` failed Job remains historical evidence only. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.
@@ -146,7 +146,7 @@ Next: <single next action>
| P1-011 | DONE | 100 | Confirm 2026-06-12 backup convergence | 18:55 live check confirms the post-120 aggregate held: no stale jobs, no configured/missing script jobs, no failed components, offsite fresh, and only credential escrow remains as DR warning. | Keep escrow as explicit red gate. | `stale110=none`, `stale188=none`, `failed=0`, `config_failed=0`, `core_blockers=0`. |
| P1-012 | DONE | 100 | Audit credential escrow marker write safety | 2026-06-12 15:02 `mark-credential-escrow-verified.sh --status` reports all five allowed items missing; `offsite-escrow-evidence-report.sh --no-color` reports rclone/offsite configured and `ESCROW_MISSING_COUNT=5`; repo search found only runbooks/placeholders/rules, not real evidence IDs. | Write markers only after a real non-secret evidence ID exists for each item; never write placeholder or secret. | The marker blocker is narrowed to missing external evidence IDs, not missing script/config/offsite readiness. |
| P1-014 | DONE | 100 | Publish credential escrow owner request package | 2026-06-13 13:10 live report confirms `SCRIPT_MISSING_COUNT=0`, `OFFSITE_CONFIGURED=1`, `RCLONE_CONFIGURED=1`, `ESCROW_MISSING_COUNT=5`, `PASS=8 WARN=5 BLOCKED=0`. New owner request package defines allowed evidence-id types, forbidden secret values, safe dry-run flow, write flow, and closeout gates. | Dispatch to the credential owners without collecting secret values; keep marker write gated until owner gives real non-secret evidence IDs. | `docs/security/CREDENTIAL-ESCROW-EVIDENCE-OWNER-REQUEST.md` and snapshot exist and validate. |
| P1-013 | IN_PROGRESS | 97 | Remediate `km-vectorize` CronJob health debt | ArgoCD Degraded is isolated to `CronJob/km-vectorize`. The schedule is correct (`0 3` with `timeZone: Asia/Taipei`), evidence retention is live, and source/log audit found a strong root-cause candidate: the internal CronJob called `/api/v1/knowledge/embed-all` without `X-Project-ID`, while `get_db_context()` now fail-closes without project context. Deploy marker `ec03f0b7` put tenant-context patch image `8ddb80d6` live and CronJob has `KM_PROJECT_ID=awoooi`. | Verify the next 03:00 official CronJob either succeeds or leaves inspectable failed Pod/log evidence. Do not manual-run or patch live. | `lastSuccessfulTime` is after the manifest sync, the official scheduled Job is `Complete`, and ArgoCD `awoooi-prod` health is `Healthy`; if it fails, the failed Job/Pod/log remains available for read-only triage. |
| P1-013 | DONE_FOR_SERVICE_READINESS | 100 | Remediate `km-vectorize` CronJob health debt | The retained `km-vectorize-29689620` failed Job is now classified as stale evidence, not an active blocker, because later official `km-vectorize` Jobs completed successfully. 2026-06-18 13:43 cold-start reads `FAILED_JOBS=1`, `STALE_FAILED_JOBS=1`, `ACTIVE_FAILED_JOBS=0`, `BAD_PODS=0`, and returns `PASS=84 WARN=0 BLOCKED=0`. | Keep retained failed Job as evidence unless an explicit maintenance window authorizes cleanup. Reassert ArgoCD app health only with a fresh ArgoCD app readback, not from the cold-start scorecard alone. | Service readiness no longer warns on stale failed Job evidence; active failed Job detection remains guarded. |
---
@@ -175,7 +175,7 @@ Next: <single next action>
| P3-005 | DONE | 100 | Update cold-start SOP | SOP now includes start, shutdown, reboot, record, comparison, and 120 blocker handling. | Increment SOP version after each process change. | SOP has controlled power-operation sections and ledger template. |
| P3-006 | DONE | 100 | Update backup status | Backup status now reflects current cron, rclone latest-only, failure-only alert posture, and escrow blocker. | Refresh after 120 backup rerun. | Backup status no longer claims noisy success Telegram notifications. |
| P3-007 | DONE | 100 | Harden Gitea backup stale dump handling | 2026-06-05 manual Gitea backup failed because the container retained `/tmp/gitea-dump.zip` from the 02:00 failure. `scripts/backup/backup-gitea.sh` now renames stale container dump files to timestamped evidence before running a new dump, and the live 110 script is updated. | Watch the next 02:00 Gitea backup. | `bash -n` passes locally and on 110; manual Gitea backup completed after stale evidence rename. |
| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.24 adds startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline, K3s filesystem event blocker, repo-side readiness audit blocker closure, 2026-06-18 live cold-start readback, post-reboot / post-CD recovery anchors, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note，以及 allowed declaration wording. | Use v1.24 for the next reboot record, then compare actual timing, Plan B trigger, degraded level, and blockers against §1.4 plus §14.8 through §14.24. Before any real reboot, rerun same-day live cold-start / backup / offsite / alert / escrow checks. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN`, `K3S_CONTROL_PLANE_AA`, `WORKLOAD_BALANCED`, `B0_ABORTED_BEFORE_REBOOT`, `B1_HOST_RECOVERY_ONLY`, `B2_CORE_SERVICE_READY`, `B3_SERVICE_AVAILABLE_DEGRADED`, `B4_FULL_STACK_GREEN`, and `B5_DR_COMPLETE`; repo-side `reboot-recovery-readiness-audit.sh --no-color` returns `PASS=185 WARN=1 BLOCKED=0`, and live cold-start returns `PASS=83 WARN=1 BLOCKED=0` with one retained stale km-vectorize failed Job warning. |
| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.25 adds startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline, K3s filesystem event blocker, repo-side readiness audit blocker closure, stale-vs-active K8s failed Job classification, 2026-06-18 live cold-start GREEN readback, post-reboot / post-CD recovery anchors, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note，以及 allowed declaration wording. | Use v1.25 for the next reboot record, then compare actual timing, Plan B trigger, degraded level, failed/stale/active Job counters, and blockers against §1.4 plus §14.8 through §14.25. Before any real reboot, rerun same-day live cold-start / backup / offsite / alert / escrow checks. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN_FOR_SERVICE`, `K3S_CONTROL_PLANE_AA`, `WORKLOAD_BALANCED`, `B0_ABORTED_BEFORE_REBOOT`, `B1_HOST_RECOVERY_ONLY`, `B2_CORE_SERVICE_READY`, `B3_SERVICE_AVAILABLE_DEGRADED`, `B4_FULL_STACK_GREEN_FOR_SERVICE`, and `B5_DR_COMPLETE`; repo-side `reboot-recovery-readiness-audit.sh --no-color` returns `PASS=187 WARN=1 BLOCKED=0`, and live cold-start returns `PASS=84 WARN=0 BLOCKED=0`. |
| P3-009 | DONE | 100 | Assess 120/121 AA/AS role and host load balancing | 2026-06-12 15:19 live check confirms 120 and 121 are both `Ready control-plane`, `k3s active`, `k3s-agent inactive`, with no taints; however most AWOOOI / ArgoCD / Velero workload remains on 121 after 120 fsck recovery. New assessment defines control-plane AA vs workload AA, migration candidates from 110/188, and stateful migration blockers. | After P0 backup/offsite/cold-start green, implement topology spread for AWOOOI API/Web before moving additional services. | `docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md` exists; SOP v1.6 links AA/AS and load-balancing checks; migration implementation remains explicitly `0%`. |
| P3-010 | DONE | 100 | Update workload balancing docs with 2026-06-13 live truth | Host role assessment, workplan, SOP, backup status, and LOGBOOK are refreshed with current cold-start, backup, 188 certbot degraded, ArgoCD `km-vectorize` degraded, Gitea main `acaae999`, ArgoCD sync, and final pod placement evidence. | Keep updating this file after the next reboot or deploy. | Docs separate service-green status from DR escrow, workload rollout, and non-service governance debt. |
| P3-011 | DONE | 100 | Record `km-vectorize` remediation status | LOGBOOK, this workplan, and SOP now state the schedule/label fix, ArgoCD sync evidence, the invalid manual Job boundary, and the 90% waiting-for-next-schedule gate. | After next 03:00 run, update this row and the top verdict with `lastSuccessfulTime` / ArgoCD health evidence. | No document claims ArgoCD green before official CronJob success evidence exists. |
@@ -214,6 +214,14 @@ Do not run `truncate`, whole DB restore, force-push, DROP, or online root filesy
## 9. Progress Updates
```text
2026-06-18 13:43 Asia/Taipei
Phase: P1/P2/P3 live readback
Before: live cold-start was `PASS=83 WARN=1 BLOCKED=0`, result `DEGRADED`, because retained stale `km-vectorize-29689620` failed Job evidence was still counted as a service warning.
After: live cold-start is `PASS=84 WARN=0 BLOCKED=0`, result `GREEN`; P2 service readiness is now `100%`; overall recovery readiness is `99% SERVICE_GREEN_DR_ESCROW_BLOCKED`.
Evidence: `full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1`; K8s schedule counters `FAILED_JOBS=1`, `STALE_FAILED_JOBS=1`, `ACTIVE_FAILED_JOBS=0`, `BAD_PODS=0`; repo-side readiness audit `PASS=187 WARN=1 BLOCKED=0`; escrow readback `ESCROW_MISSING_COUNT=5`.
Blocked: no for full-stack service readiness. Yes for DR complete, because five credential escrow evidence markers still need real non-secret owner evidence IDs.
Next: use SOP v1.25 for the next reboot; record failed/stale/active Job counters separately; close B5 only after real credential escrow marker evidence exists.
2026-06-18 12:17 Asia/Taipei
Phase: P0/P2/P3 live readback
Before: repo-side readiness was complete, but live gate had not been rerun after the same-day push.