wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

docs(ops): 收斂重啟 repo-side readiness blockers [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-06-18 12:10:43 +08:00
parent c6e6702e88
commit 63d8361f2a
11 changed files with 255 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
docs/LOGBOOK.md
View File

@@ -75,6 +75,36 @@
- 下一段必須把 Work Items / Knowledge Base / Runs 的資產沉澱顯示整併，讓 KM、PlayBook、腳本、排程、Verifier 的 asset record 在頁面上可查，否則仍會落入「做了但使用者看不到」。
**邊界**：本輪沒有把 `NO_ACTION` 偷改成直接執行命令，沒有打開 runtime gate，沒有讀 secret，沒有發送測試 Telegram，沒有呼叫 Bot API 發正式訊息，沒有 SSH / kubectl / host write / active scan。此修正只把「無安全修復候選」從批准死結改成可追蹤處置包與資產沉澱契約；真正自動修復仍必須經修復候選、owner review、approval gate、executor、verifier 與 KM / PlayBook writeback。
## 2026-06-18｜重啟 repo-side readiness audit hard blockers 收斂
**背景**：Plan B 已寫入 SOP / baseline / readiness audit，但上一輪完整 readiness audit 仍顯示大量既有 repo-side `BLOCKED`，包含 K3s filesystem event gate、AWOOOI backup direct offsite sync、110 / 188 Ansible source-of-truth、Gitea workflow triggers、PyYAML / Ansible validation toolchain、backup alert label contract 與 recovery scorecard contract。統帥要求不要停在「差一點」，因此本輪先把可 repo-only 修掉的 hard blockers 收斂到可機器驗證。
**完成內容**：
- `scripts/reboot-recovery/full-stack-cold-start-check.sh` 新增 `NODE_FS_ERROR_EVENTS`：若 K3s Node event 出現 filesystem / fsck / read-only / I/O 類證據，cold-start 會阻擋 K3s safe 宣告。
- `scripts/backup/backup-awoooi.sh` 移除 service-level 直接 offsite sync；AWOOOI DB backup 不再自行跑直連 rclone，同步交由 gated `sync-offsite-backups.sh` / verifier。
- `infra/ansible/playbooks/110-devops.yml` 納入 cold-start monitor、runner guardrails、host textfile exporters、backup scripts、daily backup heartbeat、offsite evidence report 與 offsite full-sync verifier。
- `infra/ansible/playbooks/188-ai-web.yml` 納入 textfile exporters，momo PostgreSQL backup 固定使用 host-owned `/home/ollama/bin/momo-pg-backup.sh`，並移除舊 app-directory cron path。
- `infra/ansible/playbooks/nginx-sync.yml` 納入 `188-internal-tools-https.conf.j2` source-of-truth。
- `.gitea/workflows/ansible-lint.yml` 改為 self-hosted validation workflow，觸發範圍包含 Ansible、ops baseline、monitoring rules、backup scripts、reboot scripts、docs 與 workflow 自身。
- `scripts/ops/bootstrap-ansible-validation-env.sh` 優先使用 Python 3.11 / 3.10 建立 pinned Ansible validation venv；`scripts/ops/ansible-validate.sh` 固定 repo roles path，並以 minimum lint profile守住 reboot readiness 所需的 syntax / loader safety。
- `docs/runbooks/FULL-STACK-COLD-START-SOP.md` 升為 `v1.23`，新增 repo-side readiness audit blocker closure 錨點；workplan 已同步。
**驗證**：
- `bash scripts/ops/bootstrap-ansible-validation-env.sh --recreate` 成功，使用 `python3.11` 與 `ansible-core 2.17.14`。
- `PATH=/tmp/awoooi-ansible-venv/bin:$PATH bash scripts/ops/ansible-validate.sh` 通過：YAML / Shell / Python / doc secrets / backup alert label / recovery scorecard / Ansible syntax-check / ansible-lint minimum profile 全過。
- `PATH=/tmp/awoooi-ansible-venv/bin:$PATH bash scripts/reboot-recovery/reboot-recovery-readiness-audit.sh --no-color` 回 `PASS=185 WARN=1 BLOCKED=0`，結果 `READY WITH WARNINGS`；唯一 warning 是 live cold-start gate skipped。
- `python3 scripts/ops/doc-secrets-sanity-check.py docs .gitea`：`DOC_SECRET_SANITY_OK scanned_files=898`。
- `git diff --check` 通過。
**完成度同步**：
- Repo-side reboot readiness hard blockers：`0% -> 100%`。
- Reboot SOP / Plan B / automation contracts：`100%`。
- Live reboot authorization：仍需當日 live preflight，不得由 repo-side audit 代替。
- DR complete：仍 blocked，credential escrow evidence markers 仍需 owner 提供真實非秘密 evidence ID。
**邊界**：本輪只改 repo 內 scripts / Ansible / workflow / docs，未 SSH、未套 Ansible、未重啟主機、未改 Docker / Nginx / firewall / K8s / ArgoCD、未 workflow dispatch、未讀或保存 secret、未 active scan、未送 Telegram、未改 runtime gate。
## 2026-06-18｜重啟 Plan B 機讀 baseline 與 readiness audit guard 補強
**背景**：前一段已把 Plan B 寫進 `FULL-STACK-COLD-START-SOP.md`，但如果只有 Markdown，未來仍可能在重構或同步時漂移。這輪把 Plan B 升成機讀 baseline 與 readiness audit 必檢項，讓「有沒有 Plan B」可以被工具檢查。

19
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.22
> Version: v1.23
> Last updated: 2026-06-18 Asia/Taipei
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
@@ -1464,6 +1464,23 @@ SOP update:
| Closeout states | `RETURNED_TO_PLAN_A`、`SERVICE_AVAILABLE_DEGRADED`、`OPEN_INCIDENT_REQUIRED` |
| SOP change | v1.22 新增 Plan B；不可把 Plan B 視為 runtime write 授權，也不可因文件化 Plan B 宣稱新的 service green、full-stack green 或 DR complete |
### 14.23 2026-06-18 repo-side readiness audit blocker closure
2026-06-18 的第二段變更不是 live recovery，也不是主機重啟，而是把前一輪 readiness audit 的 repo-side hard blockers 收斂成可驗證契約。這個錨點代表「重啟 SOP / baseline / scripts / Ansible source-of-truth / Gitea workflow contract 在 repo 內已可通過 readiness audit」，不代表當日 live hosts 已重新驗證。
| 項目 | 2026-06-18 repo-side readiness baseline |
|------|-----------------------------------------|
| SOP version | `v1.23` |
| Cold-start gate | `full-stack-cold-start-check.sh` 新增 `NODE_FS_ERROR_EVENTS`，120 / K3s node event 出現 filesystem / fsck / read-only / I/O 類證據時，不能宣稱 K3s safe |
| Backup contract | `backup-awoooi.sh` 移除 service-level 直接 offsite sync；offsite 發布只走集中 `sync-offsite-backups.sh` / verifier gate |
| Ansible 110 source-of-truth | `110-devops.yml` 納入 cold-start monitor、runner guardrails、host textfile exporters、backup scripts、daily backup heartbeat、offsite evidence report、offsite full-sync verifier |
| Ansible 188 source-of-truth | `188-ai-web.yml` 納入 textfile exporters，並把 momo PostgreSQL backup entrypoint 固定到 host-owned `/home/ollama/bin/momo-pg-backup.sh` |
| Nginx source-of-truth | `nginx-sync.yml` 納入 `188-internal-tools-https.conf.j2` route sync |
| CI / workflow contract | `.gitea/workflows/ansible-lint.yml` 改為 self-hosted validation，觸發範圍包含 Ansible、ops baseline、monitoring rules、backup scripts、reboot scripts、docs 與 workflow 自身 |
| Validation toolchain | `bootstrap-ansible-validation-env.sh` 會優先使用 Python 3.11 / 3.10 建立 pinned validation venv；`ansible-validate.sh` 固定 repo roles path，並以 minimum lint profile 守住 syntax / loader readiness |
| Repo-side readiness audit | `PASS=185 WARN=1 BLOCKED=0`，結果 `READY WITH WARNINGS`；唯一 warning 是未跑 `--live` |
| Declaration limit | 可宣稱 `REPO_SIDE_REBOOT_READINESS_READY_WITH_LIVE_CHECK_REQUIRED`；不可宣稱 `FULL_STACK_GREEN`、`DR_COMPLETE` 或 live service recovery complete |
### 14.22 重啟後時間軸驗證
每次重啟後照時間軸推進，不要等到最後才一次判定。

16
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -15,7 +15,7 @@
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-14 18:15 readback shows 120 is reachable, K3s is active, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-15 03:11 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-15 02:40:13`. Offsite / escrow report shows `SCRIPT_MISSING_COUNT=0`, `OFFSITE_CONFIGURED=1`, `RCLONE_CONFIGURED=1`, `ESCROW_MISSING_COUNT=5`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. |
| P2 service / data truth | VERIFIED_ARGOCD_HEALTHY_WITH_RESIDUAL_WARNINGS | 99% | 2026-06-15 03:11 cold-start is degraded by two warnings only; public route/API smoke is green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are mostly green, and 110 failed units remain `0`. `km-vectorize-29691060` succeeded, ArgoCD is `Healthy`, and API/Web remain split across 120 / 121. Remaining scorecard warnings are 188 momo scheduler registration/activity not confirmed and retained old K8s failed Job evidence. |
| P3 docs / automation contracts | DONE_WITH_VALIDATION_GAP | 100% | Workplan, SOP v1.22, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, post-CD no-regression readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, 高價值配置 Owner Packet 前台同步後 recovery readback，以及 `km-vectorize` official success readback 均已更新；本工作站無法執行 Ansible syntax check。 |
| P3 docs / automation contracts | REPO_SIDE_READY_WITH_LIVE_CHECK_REQUIRED | 100% | Workplan, SOP v1.23, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, 高價值配置 Owner Packet 前台同步後 recovery readback，以及 `km-vectorize` official success readback 均已更新。2026-06-18 repo-side `reboot-recovery-readiness-audit.sh --no-color` returned `PASS=185 WARN=1 BLOCKED=0`; only warning is live gate skipped. |
Full cold-start may be declared green only for the latest verified evidence set. As of 2026-06-15 03:11, `km-vectorize` and ArgoCD are healthy, but the latest scorecard is still `DEGRADED` by residual warnings. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
@@ -170,12 +170,12 @@ Next: <single next action>
|----|--------|---:|-----------|---------------|-------------|---------------|
| P3-001 | VERIFIED | 100 | Confirm hardening commit | Gitea `main` currently points to `0260ec89...`; `git merge-base --is-ancestor ae7b39d9 0260ec89...` returned true. | Keep evidence in LOGBOOK. | Gitea main contains `ae7b39d9 fix(ops): harden reboot recovery and backup alerts`. |
| P3-002 | VERIFIED | 100 | Confirm live 110 scripts | All six required scripts exist under `/home/wooo/scripts/`; cold-start script hash `31321428207308d6c159fabb679d9f1d0848194b8e6d7eb7b04a2c05779ade46` is live on 110. | Record in LOGBOOK. | Script paths and hashes recorded. |
| P3-003 | DONE | 100 | Reconcile 188 nginx Ansible baseline | Live 188 already routes `aiops.wooo.work` through VIP; the Ansible template now matches that route and has no 120 upstream for aiops. Content guard passed; `ansible-playbook` is not installed locally, so syntax-check could not be run here. | Run Ansible syntax/apply validation from the normal Ansible environment before the next route apply. | Template and live config agree; no 120 upstream for aiops. |
| P3-003 | DONE | 100 | Reconcile 188 nginx Ansible baseline | Live 188 already routes `aiops.wooo.work` through VIP; the Ansible template matches that route and has no 120 upstream for aiops. `nginx-sync.yml` now also carries the `188-internal-tools-https.conf.j2` source-of-truth path, and `ansible-validate.sh` syntax-check passes with repo-local roles path. | Run only approved dry-run/apply from the normal Ansible environment before changing live nginx. | Template and live config agree; no 120 upstream for aiops; repo-side syntax and readiness contract pass. |
| P3-004 | DONE | 100 | Update `docs/LOGBOOK.md` | Live blocker and new docs are recorded. | Keep this entry updated after each recovery phase. | LOGBOOK has current recovery status and next actions. |
| P3-005 | DONE | 100 | Update cold-start SOP | SOP now includes start, shutdown, reboot, record, comparison, and 120 blocker handling. | Increment SOP version after each process change. | SOP has controlled power-operation sections and ledger template. |
| P3-006 | DONE | 100 | Update backup status | Backup status now reflects current cron, rclone latest-only, failure-only alert posture, and escrow blocker. | Refresh after 120 backup rerun. | Backup status no longer claims noisy success Telegram notifications. |
| P3-007 | DONE | 100 | Harden Gitea backup stale dump handling | 2026-06-05 manual Gitea backup failed because the container retained `/tmp/gitea-dump.zip` from the 02:00 failure. `scripts/backup/backup-gitea.sh` now renames stale container dump files to timestamped evidence before running a new dump, and the live 110 script is updated. | Watch the next 02:00 Gitea backup. | `bash -n` passes locally and on 110; manual Gitea backup completed after stale evidence rename. |
| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.22 adds startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline, 2026-06-12 post-reboot anchor, 2026-06-13 post-CD trust/workload anchor, 2026-06-14 110 failed-unit cleanup anchor, 2026-06-14 post-CD recovery readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, 高價值配置 Owner Packet 前台同步後 recovery readback, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note，以及 allowed declaration wording. | Use v1.22 for the next reboot record, then compare actual timing, Plan B trigger, degraded level, and blockers against §1.4 plus §14.8 / §14.9 / §14.10 / §14.11 / §14.12 / §14.13 / §14.14 / §14.15 / §14.16 / §14.17 / §14.18 / §14.19 / §14.20 / §14.21 / §14.22. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN`, `K3S_CONTROL_PLANE_AA`, `WORKLOAD_BALANCED`, `B0_ABORTED_BEFORE_REBOOT`, `B1_HOST_RECOVERY_ONLY`, `B2_CORE_SERVICE_READY`, `B3_SERVICE_AVAILABLE_DEGRADED`, `B4_FULL_STACK_GREEN`, and `B5_DR_COMPLETE`; `reboot-recovery-readiness-audit.sh` now blocks if SOP / baseline lose these Plan B guardrails. |
| P3-008 | DONE | 100 | Continuously optimize host reboot SOP | SOP v1.23 adds startup judgment layers, GO/NO-GO decision tree, freeze execution checklist, host boot detection, 110/188/120/121 recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline, K3s filesystem event blocker, repo-side readiness audit blocker closure, 2026-06-12 post-reboot anchor, 2026-06-13 post-CD trust/workload anchor, 2026-06-14 110 failed-unit cleanup anchor, 2026-06-14 post-CD recovery readback, P2-135 deploy recovery readback, P2-136 / AI Agent 活動正式部署後 recovery readback, P2-137 / CI smoke timeout recovery readback, P2-143 owner response 預檢後 recovery readback, P2-144 owner response 回讀後 recovery readback, P2-145 owner response 驗收門檻後 recovery readback, IwoooS P0 配置控管優先序後 recovery readback, 高價值配置 Owner Packet 前台同步後 recovery readback, AA/AS 判定, workload 分散判定, CD SSH trust guardrail, CronJob failure evidence retention rule, `fwupd-refresh.timer` rollback note，以及 allowed declaration wording. | Use v1.23 for the next reboot record, then compare actual timing, Plan B trigger, degraded level, and blockers against §1.4 plus §14.8 through §14.23. Before any real reboot, rerun same-day live cold-start / backup / offsite / alert / escrow checks. | SOP distinguishes `HOST_BOOTED`, `HOST_READY`, `SERVICE_READY`, `FULL_STACK_GREEN`, `K3S_CONTROL_PLANE_AA`, `WORKLOAD_BALANCED`, `B0_ABORTED_BEFORE_REBOOT`, `B1_HOST_RECOVERY_ONLY`, `B2_CORE_SERVICE_READY`, `B3_SERVICE_AVAILABLE_DEGRADED`, `B4_FULL_STACK_GREEN`, and `B5_DR_COMPLETE`; repo-side `reboot-recovery-readiness-audit.sh --no-color` now returns `PASS=185 WARN=1 BLOCKED=0`, with the remaining warning only because live gate was intentionally skipped. |
| P3-009 | DONE | 100 | Assess 120/121 AA/AS role and host load balancing | 2026-06-12 15:19 live check confirms 120 and 121 are both `Ready control-plane`, `k3s active`, `k3s-agent inactive`, with no taints; however most AWOOOI / ArgoCD / Velero workload remains on 121 after 120 fsck recovery. New assessment defines control-plane AA vs workload AA, migration candidates from 110/188, and stateful migration blockers. | After P0 backup/offsite/cold-start green, implement topology spread for AWOOOI API/Web before moving additional services. | `docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md` exists; SOP v1.6 links AA/AS and load-balancing checks; migration implementation remains explicitly `0%`. |
| P3-010 | DONE | 100 | Update workload balancing docs with 2026-06-13 live truth | Host role assessment, workplan, SOP, backup status, and LOGBOOK are refreshed with current cold-start, backup, 188 certbot degraded, ArgoCD `km-vectorize` degraded, Gitea main `acaae999`, ArgoCD sync, and final pod placement evidence. | Keep updating this file after the next reboot or deploy. | Docs separate service-green status from DR escrow, workload rollout, and non-service governance debt. |
| P3-011 | DONE | 100 | Record `km-vectorize` remediation status | LOGBOOK, this workplan, and SOP now state the schedule/label fix, ArgoCD sync evidence, the invalid manual Job boundary, and the 90% waiting-for-next-schedule gate. | After next 03:00 run, update this row and the top verdict with `lastSuccessfulTime` / ArgoCD health evidence. | No document claims ArgoCD green before official CronJob success evidence exists. |
@@ -213,6 +213,16 @@ Do not run `truncate`, whole DB restore, force-push, DROP, or online root filesy
## 9. Progress Updates
```text
2026-06-18 12:06 Asia/Taipei
Phase: P3
Before: repo-side readiness audit PASS=147 WARN=2 BLOCKED=37 before blocker batch; after Plan B-only guard it still had pre-existing blockers.
After: repo-side readiness audit PASS=185 WARN=1 BLOCKED=0, result READY WITH WARNINGS.
Evidence: full-stack-cold-start-check.sh now emits NODE_FS_ERROR_EVENTS and blocks K3s release on node filesystem evidence; backup-awoooi.sh no longer runs direct service-level rclone sync; 110-devops.yml manages cold-start monitor, runner guardrails, textfile exporters, backup scripts, daily backup heartbeat, offsite evidence report and offsite full-sync verifier; 188-ai-web.yml uses host-owned /home/ollama/bin/momo-pg-backup.sh and no longer contains the old app-directory backup cron path; nginx-sync.yml includes 188-internal-tools-https.conf.j2; ansible-lint.yml now runs self-hosted validation across Ansible, ops baseline, monitoring rules, backup scripts, reboot scripts, docs and workflow changes; bootstrap-ansible-validation-env.sh selects Python 3.11/3.10 for pinned ansible-core; ansible-validate.sh passes YAML, shell, Python, doc secret, backup alert label, recovery scorecard, Ansible syntax-check and ansible-lint minimum profile.
Blocked: no for repo-side reboot readiness contracts. Yes for live reboot authorization until same-day live checks run; yes for DR complete while credential escrow evidence markers remain missing.
Next: before an actual reboot, run the same-day live preflight and then the live cold-start gate with --live or the 110 deployed monitor; do not use repo-side READY WITH WARNINGS as a substitute for host/runtime truth.
```
```text
2026-06-18 11:48 Asia/Taipei
Phase: P3