wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity

docs(ops): record security mirror production closeout [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-06-13 14:12:52 +08:00
parent 834ccdba83
commit 6ef0a5605d
3 changed files with 15 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/LOGBOOK.md
View File

@@ -34046,3 +34046,12 @@ production browser smoke:
- `6cf8d3ca` 修正了 Web messages，但後續空的 `ecd4531a chore(cd): trigger P2-107 production deploy` 沒有 `apps/**` 變更，導致 ArgoCD source revision 前進但 Web image 仍停在 `b557a4b5`。
- 新增 `apps/web/CHANGELOG.md` 記錄行以走正常 Gitea CD path，目標是重建 Web image，讓 security mirror messages 真正進 production bundle。
- 不手動 `kubectl apply`、不手動 ArgoCD sync、不改 runtime secret。
**Production image closeout，14:10 Asia/Taipei**：
- Deploy marker：`2cc02f1c chore(cd): deploy 6cf8d3c [skip ci]`。
- ArgoCD：revision `64ea244458ca87d4ea4c2e8b6fa2c39d6ad14653`，`Synced / Degraded`；Degraded 仍只由 `km-vectorize` 官方成功 gate 造成。
- Live Web image：`192.168.0.110:5000/awoooi/web:6cf8d3caa16c4895dad3c32236a8d47cb8addbd5`，已包含 security mirror messages。
- Public smoke：`/zh-TW/governance=200`、`/en/governance=200`、`/api/v1/health=healthy`。
- Guards：`SECURITY_MIRROR_PROGRESS_GUARD_OK`、`SOURCE_CONTROL_OWNER_RESPONSE_GUARD_OK`。
- Full cold-start：`PASS=83 WARN=0 BLOCKED=0`。
- 判定：security mirror source / production bundle / routes / cold-start 均已收斂；剩餘紅燈仍是 credential escrow marker 缺 `5` 個與 `km-vectorize` 下一次 03:00 官方成功 gate。

8
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -26,11 +26,11 @@
| Backup status | 13:52 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-13 02:34:06` | `GREEN_WITH_DR_ESCROW_WARNING` |
| Offsite sync / verify | 01:28 textfile: `awoooi_backup_offsite_remote_verify_ok=1`, `full_verify_fresh=1`, all 13 repos have `snapshot_count=1` and `snapshot_latest_only=1`; latest scheduled verifier log is 2026-06-12 07:20 | `GREEN` |
| Backup / cold-start alerts | 01:27 live visibility check confirms Prometheus and Alertmanager expose the 5 required credential escrow gap alerts; Prometheus rules API has all five required alert names healthy; label contract check loads 24 baseline backup alert rules | `GREEN_WITH_EXPECTED_REDLIGHTS` |
| Cold-start scorecard | 13:52 read-only scorecard after latest security mirror guard closure: `PASS=83 WARN=0 BLOCKED=0` | `GREEN` |
| Cold-start scorecard | 14:10 read-only scorecard after production Web image `6cf8d3ca` verification: `PASS=83 WARN=0 BLOCKED=0` | `GREEN` |
| momo DB parity | `4571|4571|2026-06-01|2026-06-07|2026-06-01|2026-06-07` | `GREEN` |
| momo scheduler | container healthy; scorecard reads `SCHEDULER_RECENT_ACTIVITY 1136`; detector widened and deployed to 110 | `GREEN` |
| ArgoCD app health | 13:52 ArgoCD revision `b557a4b5` is `Synced`; app remains `Degraded` only because `km-vectorize` CronJob has not completed its last execution successfully. CronJob schedule is `0 3 * * *` with `timeZone=Asia/Taipei`, `failedJobsHistoryLimit=3`, and `lastSuccessfulTime=2026-06-04T11:00:37Z`; next 03:00 run must prove success or leave inspectable failure evidence. | `GOVERNANCE_DEBT_IN_REMEDIATION` |
| Workload balancing | Latest live deployments use images from `39246c65`; API pods split `mon` / `mon1`, Web pods split `mon1` / `mon`, Worker single replica on `mon` | `GREEN` |
| ArgoCD app health | 14:10 ArgoCD revision `64ea2444` is `Synced`; app remains `Degraded` only because `km-vectorize` CronJob has not completed its last execution successfully. CronJob schedule is `0 3 * * *` with `timeZone=Asia/Taipei`, `failedJobsHistoryLimit=3`, and `lastSuccessfulTime=2026-06-04T11:00:37Z`; next 03:00 run must prove success or leave inspectable failure evidence. | `GOVERNANCE_DEBT_IN_REMEDIATION` |
| Workload balancing | Live Web image is `6cf8d3ca`, containing the security mirror messages; API/Web pods remain split across 120 / 121, Worker single replica remains healthy | `GREEN` |
| Credential escrow | 5 non-secret evidence markers missing | `BLOCKED` |
Release rule:
@@ -44,7 +44,7 @@ Do not declare DR scorecard complete while credential escrow markers are missing
2026-06-13 live rule:
```text
110 / 120 / 121 / 188 service recovery is full-stack green after the 13:52 scorecard.
110 / 120 / 121 / 188 service recovery is full-stack green after the 14:10 scorecard.
GO for controlled runner/CD release; keep AI auto-remediation governed by normal gates.
NO-GO for "DR complete" while credential escrow evidence markers are missing.
Do not fake or silence credential escrow alerts; they are the remaining correct DR red light.

3
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -11,7 +11,7 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 96% | 2026-06-13 13:52 final cold-start scorecard is `PASS=83 WARN=0 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public routes/TLS/momo DB/schedules/Alertmanager are green, API/Web remain live-verified split across 120 / 121 after security mirror guard closure, and CD no longer clobbers global `known_hosts`. 13:10 escrow report shows offsite/rclone/script readiness green, but DR remains blocked by five missing credential escrow evidence markers; ArgoCD `km-vectorize` is tracked separately as governance health debt until its official scheduled Job refreshes `lastSuccessfulTime`. |
| Overall recovery readiness | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 96% | 2026-06-13 14:10 final cold-start scorecard is `PASS=83 WARN=0 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public routes/TLS/momo DB/schedules/Alertmanager are green, Web image `6cf8d3ca` containing security mirror messages is live, API/Web remain live-verified split across 120 / 121, and CD no longer clobbers global `known_hosts`. 13:10 escrow report shows offsite/rclone/script readiness green, but DR remains blocked by five missing credential escrow evidence markers; ArgoCD `km-vectorize` is tracked separately as governance health debt until its official scheduled Job refreshes `lastSuccessfulTime`. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; host is reachable, root is mounted `rw`, failed units `0`, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-13 12:43 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`; 13:10 escrow report shows `SCRIPT_MISSING_COUNT=0`, `OFFSITE_CONFIGURED=1`, `RCLONE_CONFIGURED=1`, `ESCROW_MISSING_COUNT=5`, `PASS=8 WARN=5 BLOCKED=0`. Owner request package is now ready; actual marker write remains blocked on real non-secret evidence IDs. |
| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED | 100% | 2026-06-13 13:52 cold-start is green; public routes/TLS are green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are green. API/Web both keep 120 / 121 split placement after latest ArgoCD revision `b557a4b5`. |
@@ -59,6 +59,7 @@ Full cold-start may be declared green only for the latest verified evidence set.
| 2026-06-13 API placement hardening | IN_PROGRESS | 12:43 live refresh showed cold-start `PASS=83 WARN=0 BLOCKED=0`, but API replicas `2/2` were on 120 even though topology spread existed. Root cause: `whenUnsatisfiable=ScheduleAnyway` is a soft preference. GitOps candidate changes API/Web/Worker to `minDomains=2` + `DoNotSchedule`; completion requires ArgoCD sync, rollout readback, public route smoke, and cold-start rerun. |
| 2026-06-13 API rollout strategy hardening | LIVE_VERIFIED | First hard-spread rollout reached ArgoCD revision `17e017f5`; `DoNotSchedule` was live, but API completed with both new pods on 121 because old 120 pods were still terminating during scheduling. Second GitOps rollout reached ArgoCD revision `60f653a0`, API/Web use `maxSurge=0`, `maxUnavailable=1`, `minDomains=2`, `DoNotSchedule`, and both deployments are split `mon` / `mon1`. Public API / governance route smoke passed and 12:59 cold-start returned `PASS=83 WARN=0 BLOCKED=0`. |
| 2026-06-13 security mirror guard closure | LIVE_VERIFIED | Gitea main `b557a4b5` restores `apps/web/messages/en.json` as the required Traditional Chinese mirror of `zh-TW.json`; `security-mirror-progress-guard.py` now passes. ArgoCD revision `b557a4b5` is `Synced / Degraded` only by `km-vectorize`; API/Web/Worker are ready, API pods split `mon` / `mon1`, Web pods split `mon1` / `mon`, public API health is `healthy`, zh/en governance routes are `200`, backup status has `core_blockers=0`, and 13:52 cold-start is `PASS=83 WARN=0 BLOCKED=0`. |
| 2026-06-13 security mirror production image closeout | LIVE_VERIFIED | Gitea main `64ea2444` records the Web rebuild trigger. Deploy marker `2cc02f1c chore(cd): deploy 6cf8d3c [skip ci]` put Web image `6cf8d3ca` live; ArgoCD source revision later advanced to `64ea2444` while Web image correctly remains `6cf8d3ca` because `64ea2444` is docs/changelog only. Public `/zh-TW/governance` and `/en/governance` return `200`, API health is `healthy`, `security-mirror-progress-guard.py` passes, and 14:10 cold-start is `PASS=83 WARN=0 BLOCKED=0`. |
---