From a164c2c417e2022fa0994017954a99ca8427fee5 Mon Sep 17 00:00:00 2001 From: Your Name Date: Sat, 13 Jun 2026 00:38:18 +0800 Subject: [PATCH] docs(ops): record final cold-start evidence [skip ci] --- docs/LOGBOOK.md | 7 ++++--- docs/runbooks/FULL-STACK-COLD-START-SOP.md | 6 +++--- docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md | 2 +- ...26-06-04-reboot-cold-start-backup-recovery-workplan.md | 8 ++++---- 4 files changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 1108e8d53..51230e995 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -27,10 +27,11 @@ ## 2026-06-13｜Full-stack live refresh 與 120/121 workload balancing patch -**00:13 / 00:24 live refresh：** +**00:13 / 00:34 live refresh：** - `backup-status.sh --no-notify` 仍為核心綠：110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`escrow_missing=5`；DR 仍只卡 credential escrow evidence。 -- `full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` 於 00:24 post-rollout 回 `PASS=83 WARN=0 BLOCKED=0`，本次 evidence set 可宣稱 full-stack service green。 +- `full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` 於 00:24 post-rollout 回 `PASS=83 WARN=0 BLOCKED=0`；00:33 因 110 `known_hosts` 對 120 / 188 過期而短暫回 `PASS=58 WARN=2 BLOCKED=2`；刷新 110 trust 後，00:34 rerun 回 `PASS=83 WARN=0 BLOCKED=0`，本次 evidence set 可宣稱 full-stack service green。 - 110 / 120 / 121 / 188 ping + SSH OK；110 / 120 / 121 failed units `0`；188 `degraded` 只因 `certbot.service` 與 `snap.certbot.renew.service` failed，核心 PostgreSQL / Redis / momo / SignOz / Docker checks 綠。 +- 110 `known_hosts` 已先備份到 `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`，再只刷新 120 / 188 ED25519 host key；fingerprint：120 `SHA256:LLhtf/K09AQjRE1Csi0pDCkwvtM9/Ag48RcLgVF4EuY`，188 `SHA256:d+3QQO4Sh9hQ1DUhPEEJsxqE5D/CYVH+qTgG9TSMNaM`。 - ArgoCD `awoooi-prod` 已同步 revision `acaae99986aee2e1f5630984981ccb0f2b676bb8`；Deployment / PDB healthy，但 Application 仍 Degraded，原因是 `km-vectorize` CronJob last success 停在 `2026-06-04T11:00:37Z`。手動 Job `km-vectorize-codex-002709` 未留下可驗證完成紀錄，不能宣稱 ArgoCD 全綠。 **120 / 121 workload balancing：** @@ -38,7 +39,7 @@ - Gitea main commit `acaae999 fix(k8s): add prod workload topology spread` 只修改三個 manifest：`05-deployment-web.yaml`、`06-deployment-api.yaml`、`08-deployment-worker.yaml`。 - Strategy：`maxSkew: 1`、`topologyKey: kubernetes.io/hostname`、`whenUnsatisfiable: ScheduleAnyway`。 - Rollout 後 API 一度因新舊 Pod 交錯集中到 121；以單顆 stateless API Pod 受控重排後，最終 `awoooi-api` 與 `awoooi-web` 皆 120 / 121 各一顆，`awoooi-worker` 單副本在 121。 -- 驗證：`kubectl kustomize` / `git diff --check` 通過；public `awoooi_api=200`、`awoooi_web=307`；cold-start `PASS=83 WARN=0 BLOCKED=0`。 +- 驗證：`kubectl kustomize` / `git diff --check` 通過；public `awoooi_api=200`、`awoooi_web=307`；00:34 cold-start `PASS=83 WARN=0 BLOCKED=0`。 **完成度同步：** - Overall recovery readiness 維持 `95% SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED`。 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 038b3772c..865bbc49b 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -22,11 +22,11 @@ | 121 reachability | ping OK, SSH OK, failed units `0` | `GREEN` | | 188 host runtime | production services green; host degraded only by `certbot.service` and `snap.certbot.renew.service` | `GREEN_WITH_CERTBOT_DEBT` | | K3s node state | `mon Ready control-plane`, `mon1 Ready control-plane`, no failed jobs / bad pods; API/Web are split across 120 / 121 | `GREEN_WORKLOAD_BALANCED` | -| 110 -> 188 SSH trust | 188 ED25519 fingerprint matched `SHA256:d+3QQO4Sh9hQ1DUhPEEJsxqE5D/CYVH+qTgG9TSMNaM`; 110 known_hosts refreshed after reboot and SSH read-only check succeeds | `GREEN` | +| 110 -> 120 / 188 SSH trust | 00:33 cold-start exposed stale `known_hosts`; backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; 120 ED25519 `SHA256:LLhtf/K09AQjRE1Csi0pDCkwvtM9/Ag48RcLgVF4EuY`; 188 ED25519 `SHA256:d+3QQO4Sh9hQ1DUhPEEJsxqE5D/CYVH+qTgG9TSMNaM`; strict SSH checks now pass | `GREEN` | | Backup status | 00:13 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5` | `GREEN_WITH_DR_ESCROW_WARNING` | | Offsite sync / verify | 17:37 full sync `13/13`; 18:55 verifier: 13 repos each `snapshots=1`, `REMOTE_LATEST_ONLY_OK=1`, `FULL_MARKER_FRESH=1`, `VERIFY_OK=1`, `FAILED=0` | `GREEN` | | Backup / cold-start alerts | 14:49 Prometheus and Alertmanager expose 5 required backup/cold-start/escrow alerts | `GREEN_WITH_EXPECTED_REDLIGHTS` | -| Cold-start scorecard | 00:24 post-rollout read-only scorecard: `PASS=83 WARN=0 BLOCKED=0` | `GREEN` | +| Cold-start scorecard | 00:34 final read-only scorecard: `PASS=83 WARN=0 BLOCKED=0` | `GREEN` | | momo DB parity | `4571|4571|2026-06-01|2026-06-07|2026-06-01|2026-06-07` | `GREEN` | | momo scheduler | container healthy; scorecard reads `SCHEDULER_RECENT_ACTIVITY 1136`; detector widened and deployed to 110 | `GREEN` | | ArgoCD app health | Deployments/PDB/CronJobs mostly healthy; app Degraded because `km-vectorize` last success is stale | `GOVERNANCE_DEBT` | @@ -44,7 +44,7 @@ Do not declare DR scorecard complete while credential escrow markers are missing 2026-06-13 live rule: ```text -110 / 120 / 121 / 188 service recovery is full-stack green after the 00:24 scorecard. +110 / 120 / 121 / 188 service recovery is full-stack green after the 00:34 scorecard. GO for controlled runner/CD release; keep AI auto-remediation governed by normal gates. NO-GO for "DR complete" while credential escrow evidence markers are missing. Do not fake or silence credential escrow alerts; they are the remaining correct DR red light. diff --git a/docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md b/docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md index 30b9c311d..401b49033 100644 --- a/docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md +++ b/docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md @@ -187,7 +187,7 @@ sudo kubectl top pods -A --sort-by=cpu - `awoooi-api`：120 / 121 各一顆。 - `awoooi-web`：120 / 121 各一顆。 - `awoooi-worker`：單副本在 121，符合單副本判定。 -- Rollout 後冷啟動：2026-06-13 00:24 scorecard `PASS=83 WARN=0 BLOCKED=0`。 +- Rollout 後冷啟動：2026-06-13 00:34 final scorecard `PASS=83 WARN=0 BLOCKED=0`。00:33 曾因 110 `known_hosts` 對 120 / 188 過期而短暫 blocked，已在備份後刷新 host key trust。 補充：rolling update 當下 API 新 Pod 先集中到 121，因為舊 Pod 正在 120 Terminating；後續以單顆 stateless API Pod 受控重排，最終達到 120 / 121 各一顆。沒有執行全 namespace delete、Docker restart、Nginx reload 或 firewall 變更。 diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index bad885fe2..318df561e 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -11,7 +11,7 @@ | Area | Status | Completion | Evidence | |------|--------|------------|----------| -| Overall recovery readiness | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 95% | 2026-06-13 00:24 post-rollout cold-start scorecard is `PASS=83 WARN=0 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public routes/TLS/momo DB/schedules/Alertmanager are green, and API/Web are now spread across 120 / 121. Remaining blocker is DR-only credential escrow evidence (`escrow_missing=5`). | +| Overall recovery readiness | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 95% | 2026-06-13 00:34 final cold-start scorecard is `PASS=83 WARN=0 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public routes/TLS/momo DB/schedules/Alertmanager are green, and API/Web are now spread across 120 / 121. Remaining blocker is DR-only credential escrow evidence (`escrow_missing=5`). | | P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; host is reachable, root is mounted `rw`, failed units `0`, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. | | P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 90% | 2026-06-12 15:54 `backup-all` completed `13/13`; 17:37 offsite full sync completed `13/13`; 18:55 verifier returned `REMOTE_LATEST_ONLY_OK=1`, `VERIFY_OK=1`, `FAILED=0`; 18:55 backup-status has `core_blockers=0` but `escrow_missing=5`. | | P2 service / data truth | VERIFIED_WORKLOAD_BALANCED | 100% | 2026-06-13 post-rollout cold-start is green; public routes/TLS are green, VIP API/Web are reachable, momo current-month parity is `4571/4571` with matching date bounds, schedules/services are green. API/Web now both have 120 / 121 split placement. | @@ -19,7 +19,7 @@ Full cold-start may be declared green only for the latest verified evidence set. Do not declare DR scorecard complete while credential escrow evidence remains blocked. -2026-06-13 00:24 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing is live verified after Gitea main `acaae999` and ArgoCD sync; do not declare DR complete while credential escrow is missing. +2026-06-13 00:34 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing is live verified after Gitea main `acaae999` and ArgoCD sync; do not declare DR complete while credential escrow is missing. --- @@ -53,7 +53,7 @@ Full cold-start may be declared green only for the latest verified evidence set. | 2026-06-12 post-reboot recovery | SERVICE_GREEN_WITH_120_BLOCKER | 14:47 scorecard: `PASS=72 WARN=2 BLOCKED=3`; 110 failed units `0`, Swap `0B`, public routes/TLS green, momo scheduler and DB parity green, backup/offsite/alert surfaces green except the correct 120 config capture and escrow evidence red gates. | | 2026-06-12 blocker pursuit | WAITING_EXTERNAL_ACCESS | 15:00 four-view 120 check still failed; no WOL/IPMI/vmrun/hypervisor entry found in repo, 110, 121, 188, local tools, or Chronicle-visible console. 15:02 escrow report shows offsite ready with warnings and all five escrow markers missing; no real non-secret evidence ID found in repo. | | 2026-06-12 120 recovery closeout | SERVICE_GREEN_DR_ESCROW_BLOCKED | 120 root fsck was completed from console/initramfs and booted at `15:13`; 15:54 backup-all finished `13/13`; 17:37 full offsite sync finished `13/13`; 18:55 offsite verifier returned `REMOTE_LATEST_ONLY_OK=1`, `VERIFY_OK=1`, `FAILED=0`; 18:55 backup-status shows `core_blockers=0`, `escrow_missing=5`; 18:57 cold-start is `PASS=83 WARN=0 BLOCKED=0`. | -| 2026-06-13 live refresh | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 00:13 backup status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`; 00:24 post-rollout cold-start: `PASS=83 WARN=0 BLOCKED=0`; live K3s has `mon` / `mon1` Ready, API/Web are split 120 / 121. 188 host is `degraded` only because `certbot.service` and `snap.certbot.renew.service` failed; ArgoCD remains Degraded because `km-vectorize` CronJob last success is stale. Manual Job `km-vectorize-codex-002709` did not leave verified completion evidence, so this remains open. | +| 2026-06-13 live refresh | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 00:13 backup status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`; 00:33 cold-start exposed 110 `known_hosts` drift for 120 / 188, fixed after backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; 00:34 final cold-start: `PASS=83 WARN=0 BLOCKED=0`; live K3s has `mon` / `mon1` Ready, API/Web are split 120 / 121. 188 host is `degraded` only because `certbot.service` and `snap.certbot.renew.service` failed; ArgoCD remains Degraded because `km-vectorize` CronJob last success is stale. Manual Job `km-vectorize-codex-002709` did not leave verified completion evidence, so this remains open. | --- @@ -138,7 +138,7 @@ Next: | P2-004 | DONE | 100 | PostgreSQL index corruption runbook path | SOP v1.2 now states `posting list tuple ... cannot be split` is an index repair incident. | Use only concurrent reindex if the error returns. | No truncate, no whole DB restore; `REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;` and idempotent resync evidence recorded. | | P2-005 | VERIFIED | 100 | Do not rely on route 200 only | 2026-06-12 closeout has route + DB + backup + offsite + schedule + alert + K3s + cold-start scorecard evidence. The only remaining blocker is DR credential escrow, outside service availability. | Keep this cross-surface checklist mandatory after every reboot. | Each reboot record has route, DB, backup, schedules, alert, scorecard rows. | | P2-006 | DONE | 100 | Validate momo scheduler WARN | 2026-06-12 post-reboot regression showed the old detector was too narrow for Chinese batch and `[Feeder]` logs. The detector was widened and deployed to 110; 14:47 scorecard reads `SCHEDULER_RECENT_ACTIVITY 1070` and marks scheduler healthy. | Keep normal monitoring; treat future recurrence as detector tuning only if direct logs remain active. | Container healthy, direct log activity exists, and latest scorecard removed this WARN. | -| P2-007 | DONE | 100 | Balance K3s AWOOI workload across 120 / 121 | Gitea main `acaae999` adds topology spread for API/Web/Worker. ArgoCD synced revision `acaae99986aee2e1f5630984981ccb0f2b676bb8`, live deployments have non-empty `topologySpreadConstraints`, API/Web are each split 120 / 121, and 00:24 cold-start is `PASS=83 WARN=0 BLOCKED=0`. | Watch the next normal deploy to confirm topology spread remains effective. | Live deployment has non-empty topology spread, API/Web placement max skew <= 1, public routes green, cold-start `WARN=0 BLOCKED=0`. | +| P2-007 | DONE | 100 | Balance K3s AWOOI workload across 120 / 121 | Gitea main `acaae999` adds topology spread for API/Web/Worker. ArgoCD synced revision `acaae99986aee2e1f5630984981ccb0f2b676bb8`, live deployments have non-empty `topologySpreadConstraints`, API/Web are each split 120 / 121, and 00:34 final cold-start is `PASS=83 WARN=0 BLOCKED=0`. | Watch the next normal deploy to confirm topology spread remains effective. | Live deployment has non-empty topology spread, API/Web placement max skew <= 1, public routes green, cold-start `WARN=0 BLOCKED=0`. | ---