docs(ops): 記錄重啟 live readback 階段判定 [skip ci]
This commit is contained in:
Your Name
@@ -1,6 +1,6 @@
|
||||
# AWOOOI 全棧冷啟動與主機重啟 SOP
|
||||
|
||||
> Version: v1.23
|
||||
> Version: v1.24
|
||||
> Last updated: 2026-06-18 Asia/Taipei
|
||||
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
|
||||
|
||||
@@ -10,6 +10,18 @@
|
||||
|
||||
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
|
||||
|
||||
2026-06-18 12:17 live readback supersedes older service-availability wording:
|
||||
|
||||
```text
|
||||
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
|
||||
Live cold-start read-only check: PASS=83 WARN=1 BLOCKED=0, Result=DEGRADED.
|
||||
Service state: SERVICE_AVAILABLE_DEGRADED; 110/120/121/188 reachable, K3s mon/mon1 Ready, NODE_FS_ERROR_EVENTS=0, public routes/TLS green, 110/188 backup health fresh.
|
||||
Rollout state after transient 12:14 startup window: awoooi-api 2/2, awoooi-web 2/2, worker 1/1, canary 1/1, public API health 200 healthy.
|
||||
Only live warning: retained stale K8s Job km-vectorize-29689620 from 2026-06-14 03:00. Later official km-vectorize Jobs 29692500 / 29693940 / 29695380 are Complete.
|
||||
Allowed declaration: services are available with one stale failed Job warning.
|
||||
Forbidden declaration: full cold-start green, DR complete, or runtime/security acceptance.
|
||||
```
|
||||
|
||||
| 項目 | 2026-06-14 18:15 Asia/Taipei live result | 判定 |
|
||||
|------|-------------------------------------------|------|
|
||||
| Overall recovery readiness | `97%` | `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||||
@@ -55,6 +67,17 @@ NO-GO for any CD workflow that writes deploy host keys into `/home/wooo/.ssh/kno
|
||||
Current allowed wording: "core service and backup are available; 110 failed units are cleared after intentionally disabling `fwupd-refresh.timer`; high-value config Owner Packet 前台同步後 recovery readback shows no service regression; cold-start is degraded only by the `km-vectorize` official Job failure; DR complete still blocked by credential escrow; `km-vectorize` failed Job is retained but failed Pod/log are currently absent, so the next official 03:00 run remains the evidence gate."
|
||||
```
|
||||
|
||||
2026-06-18 12:17 live rule:
|
||||
|
||||
```text
|
||||
GO for controlled service availability: PASS=83 WARN=1 BLOCKED=0, public routes/TLS green, API health 200 healthy, API/Web/Worker/Canary ready after rollout convergence.
|
||||
GO for repo-side reboot readiness mechanism: readiness audit PASS=185 WARN=1 BLOCKED=0; only skipped live gate warning before the live check was run.
|
||||
NO-GO for "full cold-start green" until the retained stale failed Job evidence is either cleared by normal K8s history policy or explicitly accepted by an owner-provided readback package.
|
||||
NO-GO for "DR complete" while credential escrow evidence markers remain missing.
|
||||
Do not delete the failed Job manually during routine SOP verification. Keep it as evidence unless an approved maintenance window explicitly authorizes cleanup.
|
||||
Current allowed wording: "SOP / Plan B / automation contracts are complete; live services are available with one retained stale km-vectorize failed Job warning; hard blockers are zero; DR remains blocked by credential escrow evidence."
|
||||
```
|
||||
|
||||
After any future 120 recovery, rerun this exact chain from 110:
|
||||
|
||||
```bash
|
||||
@@ -1481,6 +1504,23 @@ SOP update:
|
||||
| Repo-side readiness audit | `PASS=185 WARN=1 BLOCKED=0`，結果 `READY WITH WARNINGS`；唯一 warning 是未跑 `--live` |
|
||||
| Declaration limit | 可宣稱 `REPO_SIDE_REBOOT_READINESS_READY_WITH_LIVE_CHECK_REQUIRED`；不可宣稱 `FULL_STACK_GREEN`、`DR_COMPLETE` 或 live service recovery complete |
|
||||
|
||||
### 14.24 2026-06-18 live cold-start readback after repo-side closure
|
||||
|
||||
2026-06-18 12:13-12:17 的 readback 是 repo-side readiness closure 後的同日 live 驗證。這不是主機重啟，也不是 runtime 修復；它的用途是把「機制已完成」和「當下 live 狀態」分開，避免 false-green。
|
||||
|
||||
| 項目 | 2026-06-18 12:17 live baseline |
|
||||
|------|--------------------------------|
|
||||
| SOP version | `v1.24` |
|
||||
| Cold-start read-only result | `PASS=83 WARN=1 BLOCKED=0`，result `DEGRADED` |
|
||||
| Host reachability | 110 / 120 / 121 / 188 ping OK and SSH port OK |
|
||||
| K3s | `mon` / `mon1` Ready control-plane；VIP `192.168.0.125` present on 120；`NODE_FS_ERROR_EVENTS 0` |
|
||||
| 110 / 188 service checks | 110 Harbor / Gitea / Prometheus / Alertmanager / Sentry reachable；188 PostgreSQL / Redis / momo / SigNoz reachable |
|
||||
| Backup health | 110 backup health `total=13 stale=0 missing_cron=0 missing_script=0 failed_count=0 config_failed=0 integrity_total=2 integrity_stale=0`；188 backup health `total=2 stale=0` |
|
||||
| Public route / TLS | awoooi API/Web、mo、momo health、Gitea、Harbor、registry、Sentry、SigNoz、stock、Langfuse、Bitan、aiops all 2xx/3xx with TLS verified |
|
||||
| AWOOOI rollout convergence | After transient 12:14 startup window, final readback shows API `2/2`, Web `2/2`, Worker `1/1`, Canary `1/1`, API health `200 healthy` |
|
||||
| Remaining warning | retained stale Job `km-vectorize-29689620` from 2026-06-14 03:00; later official Jobs `km-vectorize-29692500`, `29693940`, `29695380` are `Complete` |
|
||||
| Declaration limit | 可宣稱 `SERVICE_AVAILABLE_DEGRADED`；不可宣稱 `FULL_STACK_GREEN`，因為 `WARN=1`；不可宣稱 `DR_COMPLETE`，credential escrow evidence still requires real non-secret owner evidence |
|
||||
|
||||
### 14.22 重啟後時間軸驗證
|
||||
|
||||
每次重啟後照時間軸推進，不要等到最後才一次判定。
|
||||
|
||||
Reference in New Issue
Block a user