diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 688af7bd5..1ab144f79 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -34021,3 +34021,18 @@ production browser smoke: - `python3 scripts/security/security-mirror-progress-guard.py --root .`：`SECURITY_MIRROR_PROGRESS_GUARD_OK`。 - `pnpm --filter @awoooi/web typecheck`：通過。 - `git diff --check`、`doc-secrets-sanity-check.py docs .gitea`、`source-control-owner-response-guard.py --root .`：通過。 + +**Post-merge live verification，13:52 Asia/Taipei**： +- Gitea main：`b557a4b5 fix(web): restore en security mirror messages`。 +- ArgoCD `awoooi-prod`：`Synced / Degraded`，revision `b557a4b53ea9f95a125c7b598a57a1c531b274ed`；Degraded 仍只由 `km-vectorize` success gate 造成。 +- Deployment readback：API `2/2`、Web `2/2`、Worker `1/1`，image `39246c65`；API pods split `mon` / `mon1`，Web pods split `mon1` / `mon`。 +- `km-vectorize`：`schedule=0 3 * * *`、`timeZone=Asia/Taipei`、`failedJobsHistoryLimit=3`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；下一個 03:00 官方排程仍是 ArgoCD fully healthy gate。 +- Public smoke：`/zh-TW/governance=200`、`/en/governance=200`、`/api/v1/health=healthy`。 +- Backup status：110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`escrow_missing=5`、last aggregate `2026-06-13 02:34:06`。 +- Full cold-start：`PASS=83 WARN=0 BLOCKED=0`，result `GREEN`。 + +**目前判定**： +- Service / cold-start / workload balancing：`GREEN`。 +- Security mirror guard：`GREEN`。 +- DR scorecard：仍 `BLOCKED`，credential escrow marker 缺 `5` 個。 +- ArgoCD fully healthy：仍等待 `km-vectorize` 下次官方排程成功更新 `lastSuccessfulTime`。 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index e3f109a32..b8fc368a2 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -12,7 +12,7 @@ | 項目 | 2026-06-13 01:29 Asia/Taipei live result | 判定 | |------|-------------------------------------------|------| -| Overall recovery readiness | `95%` | `SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED` | +| Overall recovery readiness | `96%` | `SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED` | | P0 host / K3s recovery | `100%` | `DONE` | | P1 backup / alert / escrow | `90%` | `BLOCKED_DR_ESCROW` | | P2 service / data truth | `100%` | `VERIFIED_WORKLOAD_BALANCED` | @@ -23,14 +23,14 @@ | 188 host runtime | production services green; host degraded only by `certbot.service` and `snap.certbot.renew.service` | `GREEN_WITH_CERTBOT_DEBT` | | K3s node state | `mon Ready control-plane`, `mon1 Ready control-plane`, no failed jobs / bad pods; latest CD kept API/Web split across 120 / 121 | `GREEN_WORKLOAD_BALANCED` | | 110 -> 120 / 188 SSH trust | 00:33 cold-start exposed stale `known_hosts`; backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; final repair backup `/home/wooo/.ssh/known_hosts.before-120-188-final-refresh.20260613-011949`; CD fix `80e6ec1a` moves deploy trust to `/home/wooo/.ssh/deploy_known_hosts`; 01:28 global `known_hosts` still contains 120 / 188 and was not clobbered by deploy marker `e4a349bc` | `GREEN_WITH_GUARDRAIL` | -| Backup status | 01:26 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-12 15:54:40` | `GREEN_WITH_DR_ESCROW_WARNING` | +| Backup status | 13:52 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-13 02:34:06` | `GREEN_WITH_DR_ESCROW_WARNING` | | Offsite sync / verify | 01:28 textfile: `awoooi_backup_offsite_remote_verify_ok=1`, `full_verify_fresh=1`, all 13 repos have `snapshot_count=1` and `snapshot_latest_only=1`; latest scheduled verifier log is 2026-06-12 07:20 | `GREEN` | | Backup / cold-start alerts | 01:27 live visibility check confirms Prometheus and Alertmanager expose the 5 required credential escrow gap alerts; Prometheus rules API has all five required alert names healthy; label contract check loads 24 baseline backup alert rules | `GREEN_WITH_EXPECTED_REDLIGHTS` | -| Cold-start scorecard | 01:26 read-only scorecard after latest CD / trust guardrail: `PASS=83 WARN=0 BLOCKED=0` | `GREEN` | +| Cold-start scorecard | 13:52 read-only scorecard after latest security mirror guard closure: `PASS=83 WARN=0 BLOCKED=0` | `GREEN` | | momo DB parity | `4571|4571|2026-06-01|2026-06-07|2026-06-01|2026-06-07` | `GREEN` | | momo scheduler | container healthy; scorecard reads `SCHEDULER_RECENT_ACTIVITY 1136`; detector widened and deployed to 110 | `GREEN` | -| ArgoCD app health | 13:37 ArgoCD revision `88dc08e5` is `Synced`; app remains `Degraded` only because `km-vectorize` CronJob has not completed its last execution successfully. CronJob schedule is `0 3 * * *` with `timeZone=Asia/Taipei`; `lastScheduleTime=2026-06-12T19:00:00Z`, `lastSuccessfulTime=2026-06-04T11:00:37Z`. Source now retains failed Job evidence so the next failure is inspectable. | `GOVERNANCE_DEBT_IN_REMEDIATION` | -| Workload balancing | Latest live deployments use images from `414413a5`; API/Web each have one pod on 120 and one pod on 121 after deploy marker `e4a349bc` | `GREEN` | +| ArgoCD app health | 13:52 ArgoCD revision `b557a4b5` is `Synced`; app remains `Degraded` only because `km-vectorize` CronJob has not completed its last execution successfully. CronJob schedule is `0 3 * * *` with `timeZone=Asia/Taipei`, `failedJobsHistoryLimit=3`, and `lastSuccessfulTime=2026-06-04T11:00:37Z`; next 03:00 run must prove success or leave inspectable failure evidence. | `GOVERNANCE_DEBT_IN_REMEDIATION` | +| Workload balancing | Latest live deployments use images from `39246c65`; API pods split `mon` / `mon1`, Web pods split `mon1` / `mon`, Worker single replica on `mon` | `GREEN` | | Credential escrow | 5 non-secret evidence markers missing | `BLOCKED` | Release rule: @@ -44,7 +44,7 @@ Do not declare DR scorecard complete while credential escrow markers are missing 2026-06-13 live rule: ```text -110 / 120 / 121 / 188 service recovery is full-stack green after the 01:26 scorecard. +110 / 120 / 121 / 188 service recovery is full-stack green after the 13:52 scorecard. GO for controlled runner/CD release; keep AI auto-remediation governed by normal gates. NO-GO for "DR complete" while credential escrow evidence markers are missing. Do not fake or silence credential escrow alerts; they are the remaining correct DR red light. diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index ed6d4f682..6e46fec7a 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -11,10 +11,10 @@ | Area | Status | Completion | Evidence | |------|--------|------------|----------| -| Overall recovery readiness | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 95% | 2026-06-13 12:59 final cold-start scorecard is `PASS=83 WARN=0 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public routes/TLS/momo DB/schedules/Alertmanager are green, API/Web are live-verified split across 120 / 121 after topology strategy hardening, and CD no longer clobbers global `known_hosts`. 13:10 escrow report shows offsite/rclone/script readiness green, but DR remains blocked by five missing credential escrow evidence markers; ArgoCD `km-vectorize` is tracked separately as governance health debt until its official scheduled Job refreshes `lastSuccessfulTime`. | +| Overall recovery readiness | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 96% | 2026-06-13 13:52 final cold-start scorecard is `PASS=83 WARN=0 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public routes/TLS/momo DB/schedules/Alertmanager are green, API/Web remain live-verified split across 120 / 121 after security mirror guard closure, and CD no longer clobbers global `known_hosts`. 13:10 escrow report shows offsite/rclone/script readiness green, but DR remains blocked by five missing credential escrow evidence markers; ArgoCD `km-vectorize` is tracked separately as governance health debt until its official scheduled Job refreshes `lastSuccessfulTime`. | | P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; host is reachable, root is mounted `rw`, failed units `0`, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. | | P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-13 12:43 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`; 13:10 escrow report shows `SCRIPT_MISSING_COUNT=0`, `OFFSITE_CONFIGURED=1`, `RCLONE_CONFIGURED=1`, `ESCROW_MISSING_COUNT=5`, `PASS=8 WARN=5 BLOCKED=0`. Owner request package is now ready; actual marker write remains blocked on real non-secret evidence IDs. | -| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED | 100% | 2026-06-13 01:26 cold-start is green; public routes/TLS are green, VIP API/Web are reachable, momo current-month parity is `4571/4571` with matching date bounds, schedules/services are green. API/Web both keep 120 / 121 split placement after latest deploy marker `e4a349bc`. | +| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED | 100% | 2026-06-13 13:52 cold-start is green; public routes/TLS are green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are green. API/Web both keep 120 / 121 split placement after latest ArgoCD revision `b557a4b5`. | | P3 docs / automation contracts | DONE_WITH_VALIDATION_GAP | 100% | Workplan, SOP v1.9, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, T+0/T+60 timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, and `km-vectorize` remediation tracking are updated; Ansible syntax check is unavailable on this workstation. | Full cold-start may be declared green only for the latest verified evidence set. Do not declare DR scorecard complete while credential escrow evidence remains blocked. @@ -58,6 +58,7 @@ Full cold-start may be declared green only for the latest verified evidence set. | 2026-06-13 post-CD trust / workload verification | SERVICE_GREEN_CD_GUARDRAIL_HELD | Gitea main advanced to deploy marker `e4a349bc chore(cd): deploy 414413a [skip ci]`; ArgoCD revision is `e4a349bc`, sync `Synced`, health still `Degraded` only by `km-vectorize` stale success. Live K3s image readback uses `414413a59268eedd391648f112e228716dd05362`; API pods split `mon1` / `mon`, Web pods split `mon` / `mon1`, Worker is single replica on `mon`. 01:28 `/home/wooo/.ssh/known_hosts` mtime remains `2026-06-13 01:20:02 +0800` with 120 / 188 entries present; deploy-specific `/home/wooo/.ssh/deploy_known_hosts` mtime is `01:24:05`, proving CD fix `80e6ec1a` stopped clobbering global trust. 01:26 cold-start: `PASS=83 WARN=0 BLOCKED=0`. | | 2026-06-13 API placement hardening | IN_PROGRESS | 12:43 live refresh showed cold-start `PASS=83 WARN=0 BLOCKED=0`, but API replicas `2/2` were on 120 even though topology spread existed. Root cause: `whenUnsatisfiable=ScheduleAnyway` is a soft preference. GitOps candidate changes API/Web/Worker to `minDomains=2` + `DoNotSchedule`; completion requires ArgoCD sync, rollout readback, public route smoke, and cold-start rerun. | | 2026-06-13 API rollout strategy hardening | LIVE_VERIFIED | First hard-spread rollout reached ArgoCD revision `17e017f5`; `DoNotSchedule` was live, but API completed with both new pods on 121 because old 120 pods were still terminating during scheduling. Second GitOps rollout reached ArgoCD revision `60f653a0`, API/Web use `maxSurge=0`, `maxUnavailable=1`, `minDomains=2`, `DoNotSchedule`, and both deployments are split `mon` / `mon1`. Public API / governance route smoke passed and 12:59 cold-start returned `PASS=83 WARN=0 BLOCKED=0`. | +| 2026-06-13 security mirror guard closure | LIVE_VERIFIED | Gitea main `b557a4b5` restores `apps/web/messages/en.json` as the required Traditional Chinese mirror of `zh-TW.json`; `security-mirror-progress-guard.py` now passes. ArgoCD revision `b557a4b5` is `Synced / Degraded` only by `km-vectorize`; API/Web/Worker are ready, API pods split `mon` / `mon1`, Web pods split `mon1` / `mon`, public API health is `healthy`, zh/en governance routes are `200`, backup status has `core_blockers=0`, and 13:52 cold-start is `PASS=83 WARN=0 BLOCKED=0`. | ---