fix(reboot): harden public maintenance fallback
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 58s
CD Pipeline / build-and-deploy (push) Successful in 4m27s
CD Pipeline / post-deploy-checks (push) Successful in 1m44s

This commit is contained in:
Your Name
2026-07-03 04:27:54 +08:00
parent c8b421745d
commit a94ddd566e
11 changed files with 167 additions and 10 deletions

View File

@@ -20,7 +20,7 @@
| 優先 | 狀態 | 工作項 | 最新證據 | 下一步 / 完成條件 |
|------|------|--------|----------|-------------------|
| P0-1 | BLOCKED_HOST_WINDOWS | 全主機 reboot auto-detection / auto-trigger / 10 分鐘恢復 SLO | 2026-07-03 03:55 production scorecard`active_blocker_count=8``readiness_percent=67``can_claim_all_services_recovered_within_target=false``observed_host_count=7``missing_host_count=0``unreachable_host_count=1`、111 unreachable、99 uptime unknown。 | 先收斂 99 / Windows99 / VMware 與 111恢復 no-secret management channel 或 console verify stdout讀回 VMX / VM power / host uptime再 rerun host probe + reboot-event detector不得 reboot、不得 VM power change、不得讀 Windows 密碼。 |
| P0-2 | BLOCKED_PUBLIC_UX | Deploy / reboot 期間 public 502 維護頁與外部 fallback | 多次 deploy marker 切換時 SLO/API route 曾短暫 HTTP 502 且沒有 maintenance body / fallback header目前 03:55 public maintenance runtime readback 為 ready、raw 5xx count `0`,但 cutover-time 502 未被 drain / watch 消除。 | 實作 source-controlled deploy drain / maintenance fallback verifier / public route watch完成條件是 marker-time probe 不再看到 raw 502或看到明確 maintenance fallback header/body。 |
| P0-2 | IN_PROGRESS_PUBLIC_UX | Deploy / reboot 期間 public 502 維護頁與外部 fallback | Gitea CD `#4517` marker 切換時再次讀到 API / SLO route `502` 且沒有 maintenance body / fallback headersource 已改成 location-level `proxy_intercept_errors``error_page 502 503 504 =503 /__awoooi-maintenance.html`probe 也改成能把帶 fallback header/body 的 5xx 分類為 maintenance fallback。 | 推送 source hardening 後觀察 CD 與 marker-time public route watch完成條件是 marker-time probe 不再看到 raw 502或看到明確 maintenance fallback header/body。若 source 已部署但 live 仍裸 502下一步是受控 edge config readback / staging vhost / `nginx -t`,不得用 app restart 當測試。 |
| P0-3 | PARTIAL_GREEN_SOURCE_RUNTIME | 所有產品 / 網站版本與資料最新性 | Gitea `main=89d4d6112`Production SLO readback 對齊 deploy marker `89d4d6112 chore(cd): deploy 17ba08c [skip ci]`Stock freshness `status=ok``latest_trading_date=2026-07-02`、blockers `[]`。AWOOOI health HTTP 200 但整體 `degraded`SignOz / local Ollama 仍需列為 runtime degraded evidence。 | 將 source SHA / deploy marker / runtime endpoint / public route watch 固定進 scorecard完成條件是每個 public product 都有 source、deploy、runtime、freshness 四層 readback且 degraded components 有 owner lane。 |
| P0-4 | BLOCKED_WINDOWS99_AUTOSTART | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | Scorecard`windows99_update_no_auto_reboot_ready=true``windows99_vmware_verify_ready=false`、VMX missing alias `111`、powered off aliases `111/112/120/121/188`、WinRM unavailable、SSH BatchMode permission denied、RDP / Hyper-V console channel reachable。 | 只用 no-secret console / management collector 取得 `windows99-vmware-autostart.ps1 -Mode Verify` stdout先以 `validate-windows99-console-verify-artifact.py` 轉成 normalized artifact再重跑 SLO scorecard完成條件是 VMX config ready、guest power ready、99 uptime known、all required host reachable。 |
| P0-5 | PARTIAL_GREEN_BACKUP_MONITORING | Gitea / 主機 / DB / 網站 / 服務 / 套件 / 工具 / log 備份監控告警 | Gitea repo bundle readback readyexpected `12`、rows `12`、missing `0`、failed `0`、sample restore dry-run okbackup core green。這只證明 repo bundle / core backup不等於 Gitea full dump、DB/settings/issues/packages/LFS、所有工具與 log 全量備份監控完成。 | 補齊 backup health textfile / Prometheus / Telegram receipt matrix每個 backup scope 必須有 fresh、failed、age、restore-drill、alert-receipt完成條件是沒有「不知道備份有沒有跑」的盲點。 |