diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index bc50be25f..e6b407330 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -142,6 +142,27 @@ - Production browser verification:不適用;本輪是 repo 內只讀分類工具行為與文件補強,不變更前端或 runtime。 - owner response received / accepted、live evidence、runtime gate、Nginx live config、`nginx -t`、reload、DNS / TLS live probe、certbot renew、host write、active scan、production write 仍維持 `0 / false`。 +## 2026-06-14|高價值配置 Owner Packet 前台同步後 recovery readback 無回歸 + +**背景**:高價值配置 Owner Packet 前台同步已完成正式部署,feature commit 為 `e999c16b fix(iwooos): 同步高價值配置 owner packet 前台`,runtime deploy marker 為 `16c6b983 chore(cd): deploy e999c16 [skip ci]`。後續 `3de776f4`、`ddd9e433` 與 `0a4766dd` 為 `[skip ci]` 的文件 / schema / guard / snapshot 收斂;本輪只從 reboot / cold-start / backup recovery 角度做只讀 readback,不重複修改 Owner Packet 前台、posture projection、intake preflight 或 owner request draft 區塊。 + +**只讀 live readback(2026-06-14 18:15 Asia/Taipei)**: +- `backup-status.sh --no-notify`:110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`escrow_missing=5`、offsite / rclone fresh,last aggregate `2026-06-14 02:40:22`。 +- Credential escrow status 仍缺 5 個 marker:`restic_repository_password`、`offsite_provider_credentials`、`break_glass_admin_credentials`、`dns_registrar_recovery`、`oauth_ai_provider_recovery`;本輪未偽造、未寫入、未收集 secret。 +- ArgoCD `awoooi-prod` readback:sync `Synced`、health `Degraded`、source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94`;唯一 unhealthy resource 仍是 `CronJob/awoooi-prod/km-vectorize`,訊息為 `CronJob has not completed its last execution successfully`。最新 repo 文件基準為 `0a4766dd`,但 live image 基準仍由 `16c6b983` 證明。 +- K3s deployment readback:`awoooi-api` / `awoooi-web` / `awoooi-worker` / `awoooi-auto-repair-canary` 皆 Ready;API 與 Web 皆分散在 `mon` / `mon1`,Worker 單副本在 `mon`;API/Web/Worker/CronJob live image 為 `e999c16b3435f197b78fe2adfeec1c4faa6c4675`。 +- `km-vectorize` CronJob:`schedule=0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`、`failedJobsHistoryLimit=3`;failed Job `km-vectorize-29689620` 仍為 `Failed 0/1`。 +- Production API:`/api/v1/health` 回 `healthy / prod / mock_mode=false`;GCP A/B provider up,`ollama_local` 僅顯示短暫 endpoint cooldown,不列為 cold-start blocker。 +- IwoooS / AwoooP public routes:`/zh-TW/iwooos?_v=16c6b983-owner-packet-recovery-readback` 與 `/zh-TW/awooop?_v=16c6b983-owner-packet-recovery-readback` 皆回 `200`;這只證明公開頁可達,不代表 owner request sent、owner response received / accepted、runtime gate、Nginx reload、DNS / TLS probe、certbot renew、workflow / secret 修改、host write、active scan 或 production write 已開。 +- 18:15 cold-start:`PASS=82 WARN=1 BLOCKED=0`;110 / 120 / 121 / 188 reachability、188 data layer、110 registry / observability、120 / 121 K3s、AWOOOI API/Web、public routes / TLS、momo DB parity、backup exporters、CronJobs 與 120 / 121 node checks 皆通過;唯一 WARN 仍是 K8s failed Job `km-vectorize-29689620`。 + +**判讀**: +- 可以宣稱高價值配置 Owner Packet 前台同步後 recovery readback 無服務回歸;所有 public route / DB parity / backup core / K3s placement / host failed-unit gate 均維持可用。 +- 不能宣稱 full-stack green,因為 latest scorecard 仍是 `WARN=1`。 +- 不能宣稱 ArgoCD Healthy,因為 `km-vectorize` 官方 03:00 排程仍未更新 `lastSuccessfulTime`。 +- 不能宣稱 DR complete,因為 credential escrow evidence marker 仍缺 `5`。 +- 不手動建立 / 刪除 / patch `km-vectorize` Job,不重啟服務,不消音告警;下一步仍是等待下一次官方 03:00 排程,以只讀方式確認 Job / Pod / log / `lastSuccessfulTime` / ArgoCD health。 + ## 2026-06-14|IwoooS P0 配置控管優先序後 recovery readback 無回歸 **背景**:平行協作流程已完成 `/zh-TW/iwooos` P0 配置控管優先序正式驗證,最新文件基準為 `gitea/main=af62ec1f docs(iwooos): 記錄 P0 配置控管正式驗證 [skip ci]`;runtime deploy marker 為 `ed651a98 chore(cd): deploy e992af8 [skip ci]`,對應 feature commit `e992af89 feat(iwooos): 顯示 P0 配置控管優先序`。本輪只從 reboot / cold-start / backup recovery 角度做只讀 readback,不重複修改 `/zh-TW/iwooos` 同一區塊、P0 配置控管正式驗證或供應鏈進度文件。 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 20f1f9d7a..42b66541f 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,6 +1,6 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.19 +> Version: v1.20 > Last updated: 2026-06-14 Asia/Taipei > Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path. @@ -10,7 +10,7 @@ 本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天,必須先重跑 live check,再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。 -| 項目 | 2026-06-14 17:04 Asia/Taipei live result | 判定 | +| 項目 | 2026-06-14 18:15 Asia/Taipei live result | 判定 | |------|-------------------------------------------|------| | Overall recovery readiness | `97%` | `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | | P0 host / K3s recovery | `100%` | `DONE` | @@ -23,14 +23,14 @@ | 188 host runtime | production services green; host degraded only by `certbot.service` and `snap.certbot.renew.service` | `GREEN_WITH_CERTBOT_DEBT` | | K3s node state | `mon Ready control-plane`, `mon1 Ready control-plane`; latest readback keeps API/Web split across 120 / 121; bad pods `0`; retained failed Job `km-vectorize-29689620` is the only WARN | `GREEN_WORKLOAD_BALANCED_WITH_KM_WARN` | | 110 -> 120 / 188 SSH trust | 00:33 cold-start exposed stale `known_hosts`; backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; final repair backup `/home/wooo/.ssh/known_hosts.before-120-188-final-refresh.20260613-011949`; CD fix `80e6ec1a` moves deploy trust to `/home/wooo/.ssh/deploy_known_hosts`; 01:28 global `known_hosts` still contains 120 / 188 and was not clobbered by deploy marker `e4a349bc` | `GREEN_WITH_GUARDRAIL` | -| Backup status | 17:04 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22` | `GREEN_WITH_DR_ESCROW_WARNING` | +| Backup status | 18:14 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22` | `GREEN_WITH_DR_ESCROW_WARNING` | | Offsite sync / verify | 01:28 textfile: `awoooi_backup_offsite_remote_verify_ok=1`, `full_verify_fresh=1`, all 13 repos have `snapshot_count=1` and `snapshot_latest_only=1`; latest scheduled verifier log is 2026-06-12 07:20 | `GREEN` | | Backup / cold-start alerts | 01:27 live visibility check confirms Prometheus and Alertmanager expose the 5 required credential escrow gap alerts; Prometheus rules API has all five required alert names healthy; label contract check loads 24 baseline backup alert rules | `GREEN_WITH_EXPECTED_REDLIGHTS` | -| Cold-start scorecard | 17:04 IwoooS P0 配置控管優先序後只讀 scorecard:`PASS=82 WARN=1 BLOCKED=0`;remaining WARN 仍是 K8s failed Job `km-vectorize-29689620`。Public routes / TLS、momo DB parity、backup exporters、120/121 K3s 與 AWOOOI API/Web 皆通過。 | `DEGRADED_NO_BLOCKERS` | +| Cold-start scorecard | 18:15 高價值配置 Owner Packet 前台同步後只讀 scorecard:`PASS=82 WARN=1 BLOCKED=0`;remaining WARN 仍是 K8s failed Job `km-vectorize-29689620`。Public routes / TLS、momo DB parity、backup exporters、120/121 K3s 與 AWOOOI API/Web 皆通過。 | `DEGRADED_NO_BLOCKERS` | | momo DB parity | `4571|4571|2026-06-01|2026-06-07|2026-06-01|2026-06-07` | `GREEN` | -| momo scheduler | container healthy; scorecard reads `SCHEDULER_RECENT_ACTIVITY 1251`; detector widened and deployed to 110 | `GREEN` | -| ArgoCD app health | Latest live app source revision `af62ec1fe72b3e84e179d80e788e5a5902bdaf27` remains `Synced / Degraded` because official Job `km-vectorize-29689620` failed and CronJob health reports `CronJob has not completed its last execution successfully`. CronJob `lastScheduleTime=2026-06-13T19:00:00Z` and `lastSuccessfulTime=2026-06-04T11:00:37Z`; failed Job is retained, but its failed Pod/log are not currently available, so the next official 03:00 run remains the required read-only evidence gate. | `WAITING_OFFICIAL_KM_VECTORIZE_RUN` | -| Workload balancing | Live API/Web/Worker/CronJob image is `e992af89955f8aae40a383b2f2e2f645445a690d`; API/Web pods remain split across `mon` / `mon1`, Worker single replica remains healthy on `mon1` | `GREEN` | +| momo scheduler | container healthy; scorecard reads `SCHEDULER_RECENT_ACTIVITY 1009`; detector widened and deployed to 110 | `GREEN` | +| ArgoCD app health | Latest live app source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94` remains `Synced / Degraded` because official Job `km-vectorize-29689620` failed and CronJob health reports `CronJob has not completed its last execution successfully`. Latest repo docs head is `0a4766dd`, but runtime image readback is from deploy marker `16c6b983` / feature `e999c16b`. CronJob `lastScheduleTime=2026-06-13T19:00:00Z` and `lastSuccessfulTime=2026-06-04T11:00:37Z`; failed Job is retained, but its failed Pod/log are not currently available, so the next official 03:00 run remains the required read-only evidence gate. | `WAITING_OFFICIAL_KM_VECTORIZE_RUN` | +| Workload balancing | Live API/Web/Worker/CronJob image is `e999c16b3435f197b78fe2adfeec1c4faa6c4675`; API/Web pods remain split across `mon` / `mon1`, Worker single replica remains healthy on `mon` | `GREEN` | | Credential escrow | 5 non-secret evidence markers missing | `BLOCKED` | Release rule: @@ -41,10 +41,10 @@ Do not declare aggregate backup green unless latest `backup-status` has `core_bl Do not declare DR scorecard complete while credential escrow markers are missing. ``` -2026-06-14 17:04 live rule: +2026-06-14 18:15 live rule: ```text -110 / 120 / 121 / 188 core service recovery remains available, but the latest 17:04 scorecard is DEGRADED because `WARN=1`. +110 / 120 / 121 / 188 core service recovery remains available, but the latest 18:15 scorecard is DEGRADED because `WARN=1`. GO for controlled runner/CD release; keep AI auto-remediation governed by normal gates. NO-GO for "DR complete" while credential escrow evidence markers are missing. Do not fake or silence credential escrow alerts; they are the remaining correct DR red light. @@ -52,7 +52,7 @@ GO for "AWOOOI core workload balanced"; topology spread is in Gitea main / ArgoC NO-GO for "full cold-start green" until `km-vectorize` failed Job is cleared by an official successful run. NO-GO for "ArgoCD fully healthy" until `km-vectorize` updates `lastSuccessfulTime` after an official scheduled Job, not a manual `UnexpectedJob`. NO-GO for any CD workflow that writes deploy host keys into `/home/wooo/.ssh/known_hosts`; deploy jobs must use an isolated `UserKnownHostsFile=/home/wooo/.ssh/deploy_known_hosts`. -Current allowed wording: "core service and backup are available; 110 failed units are cleared after intentionally disabling `fwupd-refresh.timer`; IwoooS P0 配置控管優先序後 recovery readback shows no service regression; cold-start is degraded only by the `km-vectorize` official Job failure; DR complete still blocked by credential escrow; `km-vectorize` failed Job is retained but failed Pod/log are currently absent, so the next official 03:00 run remains the evidence gate." +Current allowed wording: "core service and backup are available; 110 failed units are cleared after intentionally disabling `fwupd-refresh.timer`; high-value config Owner Packet 前台同步後 recovery readback shows no service regression; cold-start is degraded only by the `km-vectorize` official Job failure; DR complete still blocked by credential escrow; `km-vectorize` failed Job is retained but failed Pod/log are currently absent, so the next official 03:00 run remains the evidence gate." ``` After any future 120 recovery, rerun this exact chain from 110: @@ -1330,7 +1330,25 @@ SOP update: | Remaining gate | `km-vectorize-29689620` official Job 仍 failed;Credential escrow missing count 仍 `5` | | SOP change | v1.19 記錄 IwoooS P0 配置控管優先序後 no-regression readback,並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | -### 14.19 重啟後時間軸驗證 +### 14.19 2026-06-14 高價值配置 Owner Packet 前台同步後 recovery readback + +2026-06-14 18:15 的變更不是主機重啟,而是確認高價值配置 Owner Packet 前台同步正式部署後,reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback,不重複 Owner Packet 前台正式驗證、posture projection 或 intake preflight 內容,也不把前台草案可見視為 request sent、owner response received / accepted、runtime gate、Nginx reload、DNS / TLS probe、certbot renew、workflow / secret 修改、host write、active scan 或 production write。 + +| 項目 | 2026-06-14 18:15 post-owner-packet-frontend baseline | +|------|-------------------------------------------------------| +| Gitea / ArgoCD | 最新 repo 文件基準 `0a4766dd docs(security): 新增高價值配置 owner request 草稿包 [skip ci]`;runtime deploy marker `16c6b983 chore(cd): deploy e999c16 [skip ci]`;feature commit `e999c16b fix(iwooos): 同步高價值配置 owner packet 前台`;ArgoCD source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94`,sync `Synced`,health `Degraded` | +| K3s image readback | API/Web/Worker/CronJob image tag `e999c16b3435f197b78fe2adfeec1c4faa6c4675` | +| CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`;failed Job `km-vectorize-29689620` 仍為 `Failed 0/1` | +| K3s placement | API pods split `mon` / `mon1`,Web pods split `mon` / `mon1`,Worker single replica on `mon` | +| Cold-start | 18:15 returned `PASS=82 WARN=1 BLOCKED=0` | +| Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應;IwoooS route `/zh-TW/iwooos` 與 AwoooP route `/zh-TW/awooop` 額外 readback 皆回 `200` | +| Backup | 110 `13/13 fresh failed=0`,188 `2/2 fresh failed=0`,`core_blockers=0`,`escrow_missing=5` | +| 110 host | `systemctl --failed` 回 `0 loaded units listed`;`fwupd-refresh.timer` 維持 `disabled / inactive` | +| Owner Packet boundary | Owner Packet 前台數字已可見,但 request sent、owner response received / accepted / rejected、reviewer queue write、live evidence、runtime gate、Nginx live config、DNS / TLS probe、certbot renew、workflow / secret 修改、host write、active scan、production write 仍不得從本 readback 推定為已授權 | +| Remaining gate | `km-vectorize-29689620` official Job 仍 failed;Credential escrow missing count 仍 `5` | +| SOP change | v1.20 記錄高價值配置 Owner Packet 前台同步後 no-regression readback,並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | + +### 14.20 重啟後時間軸驗證 每次重啟後照時間軸推進,不要等到最後才一次判定。 diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index 46852a4b0..b4907578e 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -11,13 +11,13 @@ | Area | Status | Completion | Evidence | |------|--------|------------|----------| -| Overall recovery readiness | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 97% | 2026-06-14 17:04 IwoooS P0 配置控管優先序後 cold-start scorecard 為 `PASS=82 WARN=1 BLOCKED=0`;120/121 K3s 皆為 `Ready control-plane`,backup core blockers 仍為 `0`,public route/API smoke 仍綠,最新文件基準為 `af62ec1f`,runtime deploy marker 為 `ed651a98`,ArgoCD source revision 為 `af62ec1fe72b3e84e179d80e788e5a5902bdaf27`,且 API/Web/Worker/CronJob image `e992af89955f8aae40a383b2f2e2f645445a690d` 已 live,API/Web 維持分散在 120 / 121。110 `fwupd` failed-unit warning 仍已清除,`systemctl --failed` clean。Full cold-start 仍不可宣稱 green,因為官方 `km-vectorize-29689620` Job 仍 failed;DR 仍受 5 個 credential escrow evidence marker 缺口阻擋。 | -| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 17:04 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. | -| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-14 17:04 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. | -| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED_WITH_KM_WARN | 99% | 2026-06-14 17:04 cold-start is degraded by one warning only; public route/API smoke is green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are mostly green, and 110 failed units remain `0`. API/Web both keep 120 / 121 split placement after latest ArgoCD source revision `af62ec1fe72b3e84e179d80e788e5a5902bdaf27`, with live API/Web/Worker/CronJob image `e992af89955f8aae40a383b2f2e2f645445a690d`; the remaining exception is failed `km-vectorize-29689620`. | -| P3 docs / automation contracts | DONE_WITH_VALIDATION_GAP | 100% | Workplan, SOP v1.19, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, T+0/T+60 timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, post-CD no-regression readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, and `km-vectorize` remediation tracking are updated; Ansible syntax check is unavailable on this workstation. | +| Overall recovery readiness | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 97% | 2026-06-14 18:15 高價值配置 Owner Packet 前台同步後 cold-start scorecard 為 `PASS=82 WARN=1 BLOCKED=0`;120/121 K3s 皆為 `Ready control-plane`,backup core blockers 仍為 `0`,public route/API smoke 仍綠,最新 repo 文件基準為 `0a4766dd`,runtime deploy marker 為 `16c6b983`,ArgoCD source revision 為 `0a4766ddc94b0690824ce3deba5c6b9a69764f94`,且 API/Web/Worker/CronJob image `e999c16b3435f197b78fe2adfeec1c4faa6c4675` 已 live,API/Web 維持分散在 120 / 121。110 `fwupd` failed-unit warning 仍已清除,`systemctl --failed` clean。Full cold-start 仍不可宣稱 green,因為官方 `km-vectorize-29689620` Job 仍 failed;DR 仍受 5 個 credential escrow evidence marker 缺口阻擋。 | +| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. | +| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-14 18:14 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. | +| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED_WITH_KM_WARN | 99% | 2026-06-14 18:15 cold-start is degraded by one warning only; public route/API smoke is green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are mostly green, and 110 failed units remain `0`. API/Web both keep 120 / 121 split placement after latest ArgoCD source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94`, with live API/Web/Worker/CronJob image `e999c16b3435f197b78fe2adfeec1c4faa6c4675`; the remaining exception is failed `km-vectorize-29689620`. | +| P3 docs / automation contracts | DONE_WITH_VALIDATION_GAP | 100% | Workplan, SOP v1.20, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, T+0/T+60 timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, post-CD no-regression readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, 高價值配置 Owner Packet 前台同步後 recovery readback,以及 `km-vectorize` remediation tracking 均已更新;本工作站無法執行 Ansible syntax check。 | -Full cold-start may be declared green only for the latest verified evidence set. As of 2026-06-14 17:04, the latest evidence set is degraded by `km-vectorize` only, not green. Do not declare DR scorecard complete while credential escrow evidence remains blocked. +Full cold-start may be declared green only for the latest verified evidence set. As of 2026-06-14 18:15, the latest evidence set is degraded by `km-vectorize` only, not green. Do not declare DR scorecard complete while credential escrow evidence remains blocked. 2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback. @@ -73,6 +73,7 @@ Full cold-start may be declared green only for the latest verified evidence set. | 2026-06-14 P2-144 owner response 回讀 recovery readback | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | `gitea/main` 已前進至 deploy marker `180a6543`;image `fef94df877c5438f9f34ddbcace8ad8112a141ef` 已帶到 live,ArgoCD source revision `180a6543eaf26dd6b345d45114316926056a965a` 為 `Synced / Degraded`。API/Web 維持分散在 `mon` / `mon1`,Worker 在 `mon1` healthy,backup core 仍為 `0`,escrow missing 仍為 `5`,15:58 cold-start 為 `PASS=82 WARN=1 BLOCKED=0`;P2-144 endpoint current `P2-144`、completion `100`,owner response received / accepted / rejected、reviewer / Gateway / Telegram / Bot API / result capture / learning / PlayBook trust / production write / secret read / destructive operation 維持 `0 / false`。這證明 P2-144 owner response 回讀後 recovery 沒有回歸,但仍不是 full green。 | | 2026-06-14 P2-145 owner response 驗收門檻 recovery readback | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 最新文件基準為 `06fe0a8f`;runtime deploy marker `36fbfc6b` 已將 image `386dbd078ef63401d9736048463f4ef5326442d9` 帶到 live,ArgoCD source revision `06fe0a8f14167824fea512f942d2569431bbcbc8` 為 `Synced / Degraded`。API/Web 維持分散在 `mon` / `mon1`,Worker 在 `mon` healthy,backup core 仍為 `0`,escrow missing 仍為 `5`,16:29 cold-start 為 `PASS=82 WARN=1 BLOCKED=0`;P2-145 endpoint current `P2-145`、completion `100`,owner response received / accepted / rejected、reviewer / Gateway / Telegram / Bot API / result capture / learning / PlayBook trust / production write / secret read / destructive operation 維持 `0 / false`。這證明 P2-145 owner response 驗收門檻後 recovery 沒有回歸,但仍不是 full green。 | | 2026-06-14 IwoooS P0 配置控管優先序 recovery readback | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 最新文件基準為 `af62ec1f`;runtime deploy marker `ed651a98` 已將 image `e992af89955f8aae40a383b2f2e2f645445a690d` 帶到 API/Web/Worker/CronJob live,ArgoCD source revision `af62ec1fe72b3e84e179d80e788e5a5902bdaf27` 為 `Synced / Degraded`。API/Web 維持分散在 `mon` / `mon1`,Worker 在 `mon1` healthy;IwoooS route `/zh-TW/iwooos` returned `200`。backup core 仍為 `0`,escrow missing 仍為 `5`,17:04 cold-start 為 `PASS=82 WARN=1 BLOCKED=0`。這證明 IwoooS P0 配置控管優先序前台發布後 recovery 沒有回歸;但它不代表 Nginx reload、DNS/TLS/certbot、workflow/secret/public route/runtime gate 或 production write 已授權,且仍不是 full green。 | +| 2026-06-14 高價值配置 Owner Packet 前台同步 recovery readback | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 最新 repo 文件基準為 `0a4766dd`;runtime deploy marker `16c6b983` 已將 image `e999c16b3435f197b78fe2adfeec1c4faa6c4675` 帶到 API/Web/Worker/CronJob live,ArgoCD source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94` 為 `Synced / Degraded`。API/Web 維持分散在 `mon` / `mon1`,Worker 在 `mon` healthy;IwoooS route `/zh-TW/iwooos` 與 AwoooP route `/zh-TW/awooop` 皆回 `200`。backup core 仍為 `0`,escrow missing 仍為 `5`,18:15 cold-start 為 `PASS=82 WARN=1 BLOCKED=0`。這證明高價值配置 Owner Packet 前台同步後 recovery 沒有回歸;但它不代表 request sent、owner response received / accepted、Nginx reload、DNS/TLS/certbot、workflow/secret/public route/runtime gate、host write、active scan 或 production write 已授權,且仍不是 full green。 | --- @@ -174,7 +175,7 @@ Next: | P3-005 | DONE | 100 | Update cold-start SOP | SOP now includes start, shutdown, reboot, record, comparison, and 120 blocker handling. | Increment SOP version after each process change. | SOP has controlled power-operation sections and ledger template. | | P3-006 | DONE | 100 | Update backup status | Backup status now reflects current cron, rclone latest-only, failure-only alert posture, and escrow blocker. | Refresh after 120 backup rerun. | Backup status no longer claims noisy success Telegram notifications. | | P3-007 | DONE | 100 | Harden Gitea backup stale dump handling | 2026-06-05 manual Gitea backup failed because the container retained `/tmp/gitea-dump.zip` from the 02:00 failure. `scripts/backup/backup-gitea.sh` now renames stale container dump files to timestamped evidence before running a new dump, and the live 110 script is updated. | Watch the next 02:00 Gitea backup. | `bash -n` passes locally and on 110; manual Gitea backup completed after stale evidence rename. | -| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.19 adds startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, 2026-06-12 post-reboot anchor, 2026-06-13 post-CD trust/workload anchor, 2026-06-14 110 failed-unit cleanup anchor, 2026-06-14 post-CD recovery readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, T+0/T+60 verification timeline, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note, and allowed declaration wording. | Use v1.19 for the next reboot record, then compare actual timing and blockers against §14.8 / §14.9 / §14.10 / §14.11 / §14.12 / §14.13 / §14.14 / §14.15 / §14.16 / §14.17 / §14.18 / §14.19. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN`, `K3S_CONTROL_PLANE_AA`, and `WORKLOAD_BALANCED`, and blocks false green while escrow or governed CronJob debt remain red. | +| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.20 adds startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, 2026-06-12 post-reboot anchor, 2026-06-13 post-CD trust/workload anchor, 2026-06-14 110 failed-unit cleanup anchor, 2026-06-14 post-CD recovery readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, 高價值配置 Owner Packet 前台同步後 recovery readback, T+0/T+60 verification timeline, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note,以及 allowed declaration wording. | Use v1.20 for the next reboot record, then compare actual timing and blockers against §14.8 / §14.9 / §14.10 / §14.11 / §14.12 / §14.13 / §14.14 / §14.15 / §14.16 / §14.17 / §14.18 / §14.19 / §14.20. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN`, `K3S_CONTROL_PLANE_AA`, and `WORKLOAD_BALANCED`, and blocks false green while escrow or governed CronJob debt remain red. | | P3-009 | DONE | 100 | Assess 120/121 AA/AS role and host load balancing | 2026-06-12 15:19 live check confirms 120 and 121 are both `Ready control-plane`, `k3s active`, `k3s-agent inactive`, with no taints; however most AWOOOI / ArgoCD / Velero workload remains on 121 after 120 fsck recovery. New assessment defines control-plane AA vs workload AA, migration candidates from 110/188, and stateful migration blockers. | After P0 backup/offsite/cold-start green, implement topology spread for AWOOOI API/Web before moving additional services. | `docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md` exists; SOP v1.6 links AA/AS and load-balancing checks; migration implementation remains explicitly `0%`. | | P3-010 | DONE | 100 | Update workload balancing docs with 2026-06-13 live truth | Host role assessment, workplan, SOP, backup status, and LOGBOOK are refreshed with current cold-start, backup, 188 certbot degraded, ArgoCD `km-vectorize` degraded, Gitea main `acaae999`, ArgoCD sync, and final pod placement evidence. | Keep updating this file after the next reboot or deploy. | Docs separate service-green status from DR escrow, workload rollout, and non-service governance debt. | | P3-011 | DONE | 100 | Record `km-vectorize` remediation status | LOGBOOK, this workplan, and SOP now state the schedule/label fix, ArgoCD sync evidence, the invalid manual Job boundary, and the 90% waiting-for-next-schedule gate. | After next 03:00 run, update this row and the top verdict with `lastSuccessfulTime` / ArgoCD health evidence. | No document claims ArgoCD green before official CronJob success evidence exists. | @@ -212,6 +213,16 @@ Do not run `truncate`, whole DB restore, force-push, DROP, or online root filesy ## 9. Progress Updates +```text +2026-06-14 18:15 Asia/Taipei +Phase: P0/P1/P2/P3 +Before: Overall 97%, P1 92%, P2 99%, P3 100% +After: Overall 97%, P1 92%, P2 99%, P3 100% +Evidence: 最新 repo 文件基準 0a4766dd;runtime deploy marker 16c6b983 已將 image e999c16b3435f197b78fe2adfeec1c4faa6c4675 帶到 API/Web/Worker/CronJob live;ArgoCD source revision 0a4766ddc94b0690824ce3deba5c6b9a69764f94 維持 Synced/Degraded,原因仍只剩 km-vectorize;API/Web 分散在 mon/mon1;Worker 在 mon;IwoooS route /zh-TW/iwooos returned 200;AwoooP route /zh-TW/awooop returned 200;110 systemctl --failed returned 0 loaded units listed;backup-status core_blockers=0 and escrow_missing=5;final cold-start PASS=82 WARN=1 BLOCKED=0. +Blocked: yes for full cold-start green,因為 km-vectorize-29689620 仍 failed,必須等待下一次官方 03:00 success 或 retained failed Pod/log evidence;yes for DR complete,因為 credential escrow evidence markers 仍缺 5 個。 +Next: 維持 03:00 官方排程 gate;下一次官方 km-vectorize run 後,只讀驗證 lastSuccessfulTime、latest Job/Pod/log 與 ArgoCD health。 +``` + ```text 2026-06-14 17:04 Asia/Taipei Phase: P0/P1/P2/P3