wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

docs(ops): record owner packet recovery readback [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-06-14 18:26:12 +08:00
parent 0a4766ddc9
commit 2a92087568
3 changed files with 68 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -11,13 +11,13 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 97% | 2026-06-14 17:04 IwoooS P0 配置控管優先序後 cold-start scorecard 為 `PASS=82 WARN=1 BLOCKED=0`；120/121 K3s 皆為 `Ready control-plane`，backup core blockers 仍為 `0`，public route/API smoke 仍綠，最新文件基準為 `af62ec1f`，runtime deploy marker 為 `ed651a98`，ArgoCD source revision 為 `af62ec1fe72b3e84e179d80e788e5a5902bdaf27`，且 API/Web/Worker/CronJob image `e992af89955f8aae40a383b2f2e2f645445a690d` 已 live，API/Web 維持分散在 120 / 121。110 `fwupd` failed-unit warning 仍已清除，`systemctl --failed` clean。Full cold-start 仍不可宣稱 green，因為官方 `km-vectorize-29689620` Job 仍 failed；DR 仍受 5 個 credential escrow evidence marker 缺口阻擋。 |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 17:04 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-14 17:04 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. |
| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED_WITH_KM_WARN | 99% | 2026-06-14 17:04 cold-start is degraded by one warning only; public route/API smoke is green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are mostly green, and 110 failed units remain `0`. API/Web both keep 120 / 121 split placement after latest ArgoCD source revision `af62ec1fe72b3e84e179d80e788e5a5902bdaf27`, with live API/Web/Worker/CronJob image `e992af89955f8aae40a383b2f2e2f645445a690d`; the remaining exception is failed `km-vectorize-29689620`. |
| P3 docs / automation contracts | DONE_WITH_VALIDATION_GAP | 100% | Workplan, SOP v1.19, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, T+0/T+60 timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, post-CD no-regression readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, and `km-vectorize` remediation tracking are updated; Ansible syntax check is unavailable on this workstation. |
| Overall recovery readiness | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 97% | 2026-06-14 18:15 高價值配置 Owner Packet 前台同步後 cold-start scorecard 為 `PASS=82 WARN=1 BLOCKED=0`；120/121 K3s 皆為 `Ready control-plane`，backup core blockers 仍為 `0`，public route/API smoke 仍綠，最新 repo 文件基準為 `0a4766dd`，runtime deploy marker 為 `16c6b983`，ArgoCD source revision 為 `0a4766ddc94b0690824ce3deba5c6b9a69764f94`，且 API/Web/Worker/CronJob image `e999c16b3435f197b78fe2adfeec1c4faa6c4675` 已 live，API/Web 維持分散在 120 / 121。110 `fwupd` failed-unit warning 仍已清除，`systemctl --failed` clean。Full cold-start 仍不可宣稱 green，因為官方 `km-vectorize-29689620` Job 仍 failed；DR 仍受 5 個 credential escrow evidence marker 缺口阻擋。 |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-14 18:14 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. |
| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED_WITH_KM_WARN | 99% | 2026-06-14 18:15 cold-start is degraded by one warning only; public route/API smoke is green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are mostly green, and 110 failed units remain `0`. API/Web both keep 120 / 121 split placement after latest ArgoCD source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94`, with live API/Web/Worker/CronJob image `e999c16b3435f197b78fe2adfeec1c4faa6c4675`; the remaining exception is failed `km-vectorize-29689620`. |
| P3 docs / automation contracts | DONE_WITH_VALIDATION_GAP | 100% | Workplan, SOP v1.20, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, T+0/T+60 timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, post-CD no-regression readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, 高價值配置 Owner Packet 前台同步後 recovery readback，以及 `km-vectorize` remediation tracking 均已更新；本工作站無法執行 Ansible syntax check。 |
Full cold-start may be declared green only for the latest verified evidence set. As of 2026-06-14 17:04, the latest evidence set is degraded by `km-vectorize` only, not green. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
Full cold-start may be declared green only for the latest verified evidence set. As of 2026-06-14 18:15, the latest evidence set is degraded by `km-vectorize` only, not green. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.
@@ -73,6 +73,7 @@ Full cold-start may be declared green only for the latest verified evidence set.
| 2026-06-14 P2-144 owner response 回讀 recovery readback | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | `gitea/main` 已前進至 deploy marker `180a6543`；image `fef94df877c5438f9f34ddbcace8ad8112a141ef` 已帶到 live，ArgoCD source revision `180a6543eaf26dd6b345d45114316926056a965a` 為 `Synced / Degraded`。API/Web 維持分散在 `mon` / `mon1`，Worker 在 `mon1` healthy，backup core 仍為 `0`，escrow missing 仍為 `5`，15:58 cold-start 為 `PASS=82 WARN=1 BLOCKED=0`；P2-144 endpoint current `P2-144`、completion `100`，owner response received / accepted / rejected、reviewer / Gateway / Telegram / Bot API / result capture / learning / PlayBook trust / production write / secret read / destructive operation 維持 `0 / false`。這證明 P2-144 owner response 回讀後 recovery 沒有回歸，但仍不是 full green。 |
| 2026-06-14 P2-145 owner response 驗收門檻 recovery readback | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 最新文件基準為 `06fe0a8f`；runtime deploy marker `36fbfc6b` 已將 image `386dbd078ef63401d9736048463f4ef5326442d9` 帶到 live，ArgoCD source revision `06fe0a8f14167824fea512f942d2569431bbcbc8` 為 `Synced / Degraded`。API/Web 維持分散在 `mon` / `mon1`，Worker 在 `mon` healthy，backup core 仍為 `0`，escrow missing 仍為 `5`，16:29 cold-start 為 `PASS=82 WARN=1 BLOCKED=0`；P2-145 endpoint current `P2-145`、completion `100`，owner response received / accepted / rejected、reviewer / Gateway / Telegram / Bot API / result capture / learning / PlayBook trust / production write / secret read / destructive operation 維持 `0 / false`。這證明 P2-145 owner response 驗收門檻後 recovery 沒有回歸，但仍不是 full green。 |
| 2026-06-14 IwoooS P0 配置控管優先序 recovery readback | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 最新文件基準為 `af62ec1f`；runtime deploy marker `ed651a98` 已將 image `e992af89955f8aae40a383b2f2e2f645445a690d` 帶到 API/Web/Worker/CronJob live，ArgoCD source revision `af62ec1fe72b3e84e179d80e788e5a5902bdaf27` 為 `Synced / Degraded`。API/Web 維持分散在 `mon` / `mon1`，Worker 在 `mon1` healthy；IwoooS route `/zh-TW/iwooos` returned `200`。backup core 仍為 `0`，escrow missing 仍為 `5`，17:04 cold-start 為 `PASS=82 WARN=1 BLOCKED=0`。這證明 IwoooS P0 配置控管優先序前台發布後 recovery 沒有回歸；但它不代表 Nginx reload、DNS/TLS/certbot、workflow/secret/public route/runtime gate 或 production write 已授權，且仍不是 full green。 |
| 2026-06-14 高價值配置 Owner Packet 前台同步 recovery readback | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 最新 repo 文件基準為 `0a4766dd`；runtime deploy marker `16c6b983` 已將 image `e999c16b3435f197b78fe2adfeec1c4faa6c4675` 帶到 API/Web/Worker/CronJob live，ArgoCD source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94` 為 `Synced / Degraded`。API/Web 維持分散在 `mon` / `mon1`，Worker 在 `mon` healthy；IwoooS route `/zh-TW/iwooos` 與 AwoooP route `/zh-TW/awooop` 皆回 `200`。backup core 仍為 `0`，escrow missing 仍為 `5`，18:15 cold-start 為 `PASS=82 WARN=1 BLOCKED=0`。這證明高價值配置 Owner Packet 前台同步後 recovery 沒有回歸；但它不代表 request sent、owner response received / accepted、Nginx reload、DNS/TLS/certbot、workflow/secret/public route/runtime gate、host write、active scan 或 production write 已授權，且仍不是 full green。 |
---
@@ -174,7 +175,7 @@ Next: <single next action>
| P3-005 | DONE | 100 | Update cold-start SOP | SOP now includes start, shutdown, reboot, record, comparison, and 120 blocker handling. | Increment SOP version after each process change. | SOP has controlled power-operation sections and ledger template. |
| P3-006 | DONE | 100 | Update backup status | Backup status now reflects current cron, rclone latest-only, failure-only alert posture, and escrow blocker. | Refresh after 120 backup rerun. | Backup status no longer claims noisy success Telegram notifications. |
| P3-007 | DONE | 100 | Harden Gitea backup stale dump handling | 2026-06-05 manual Gitea backup failed because the container retained `/tmp/gitea-dump.zip` from the 02:00 failure. `scripts/backup/backup-gitea.sh` now renames stale container dump files to timestamped evidence before running a new dump, and the live 110 script is updated. | Watch the next 02:00 Gitea backup. | `bash -n` passes locally and on 110; manual Gitea backup completed after stale evidence rename. |
| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.19 adds startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, 2026-06-12 post-reboot anchor, 2026-06-13 post-CD trust/workload anchor, 2026-06-14 110 failed-unit cleanup anchor, 2026-06-14 post-CD recovery readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, T+0/T+60 verification timeline, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note, and allowed declaration wording. | Use v1.19 for the next reboot record, then compare actual timing and blockers against §14.8 / §14.9 / §14.10 / §14.11 / §14.12 / §14.13 / §14.14 / §14.15 / §14.16 / §14.17 / §14.18 / §14.19. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN`, `K3S_CONTROL_PLANE_AA`, and `WORKLOAD_BALANCED`, and blocks false green while escrow or governed CronJob debt remain red. |
| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.20 adds startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, 2026-06-12 post-reboot anchor, 2026-06-13 post-CD trust/workload anchor, 2026-06-14 110 failed-unit cleanup anchor, 2026-06-14 post-CD recovery readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, 高價值配置 Owner Packet 前台同步後 recovery readback, T+0/T+60 verification timeline, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note，以及 allowed declaration wording. | Use v1.20 for the next reboot record, then compare actual timing and blockers against §14.8 / §14.9 / §14.10 / §14.11 / §14.12 / §14.13 / §14.14 / §14.15 / §14.16 / §14.17 / §14.18 / §14.19 / §14.20. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN`, `K3S_CONTROL_PLANE_AA`, and `WORKLOAD_BALANCED`, and blocks false green while escrow or governed CronJob debt remain red. |
| P3-009 | DONE | 100 | Assess 120/121 AA/AS role and host load balancing | 2026-06-12 15:19 live check confirms 120 and 121 are both `Ready control-plane`, `k3s active`, `k3s-agent inactive`, with no taints; however most AWOOOI / ArgoCD / Velero workload remains on 121 after 120 fsck recovery. New assessment defines control-plane AA vs workload AA, migration candidates from 110/188, and stateful migration blockers. | After P0 backup/offsite/cold-start green, implement topology spread for AWOOOI API/Web before moving additional services. | `docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md` exists; SOP v1.6 links AA/AS and load-balancing checks; migration implementation remains explicitly `0%`. |
| P3-010 | DONE | 100 | Update workload balancing docs with 2026-06-13 live truth | Host role assessment, workplan, SOP, backup status, and LOGBOOK are refreshed with current cold-start, backup, 188 certbot degraded, ArgoCD `km-vectorize` degraded, Gitea main `acaae999`, ArgoCD sync, and final pod placement evidence. | Keep updating this file after the next reboot or deploy. | Docs separate service-green status from DR escrow, workload rollout, and non-service governance debt. |
| P3-011 | DONE | 100 | Record `km-vectorize` remediation status | LOGBOOK, this workplan, and SOP now state the schedule/label fix, ArgoCD sync evidence, the invalid manual Job boundary, and the 90% waiting-for-next-schedule gate. | After next 03:00 run, update this row and the top verdict with `lastSuccessfulTime` / ArgoCD health evidence. | No document claims ArgoCD green before official CronJob success evidence exists. |
@@ -212,6 +213,16 @@ Do not run `truncate`, whole DB restore, force-push, DROP, or online root filesy
## 9. Progress Updates
```text
2026-06-14 18:15 Asia/Taipei
Phase: P0/P1/P2/P3
Before: Overall 97%, P1 92%, P2 99%, P3 100%
After: Overall 97%, P1 92%, P2 99%, P3 100%
Evidence: 最新 repo 文件基準 0a4766dd；runtime deploy marker 16c6b983 已將 image e999c16b3435f197b78fe2adfeec1c4faa6c4675 帶到 API/Web/Worker/CronJob live；ArgoCD source revision 0a4766ddc94b0690824ce3deba5c6b9a69764f94 維持 Synced/Degraded，原因仍只剩 km-vectorize；API/Web 分散在 mon/mon1；Worker 在 mon；IwoooS route /zh-TW/iwooos returned 200；AwoooP route /zh-TW/awooop returned 200；110 systemctl --failed returned 0 loaded units listed；backup-status core_blockers=0 and escrow_missing=5；final cold-start PASS=82 WARN=1 BLOCKED=0.
Blocked: yes for full cold-start green，因為 km-vectorize-29689620 仍 failed，必須等待下一次官方 03:00 success 或 retained failed Pod/log evidence；yes for DR complete，因為 credential escrow evidence markers 仍缺 5 個。
Next: 維持 03:00 官方排程 gate；下一次官方 km-vectorize run 後，只讀驗證 lastSuccessfulTime、latest Job/Pod/log 與 ArgoCD health。
```
```text
2026-06-14 17:04 Asia/Taipei
Phase: P0/P1/P2/P3