wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

fix(cd): keep windows99 console verifier controlled
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Successful in 59s
Details
CD Pipeline / build-and-deploy (push) Successful in 4m17s
Details
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 1s
Details
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Failing after 51s
Details
CD Pipeline / post-deploy-checks (push) Successful in 1m46s
Details

Browse Source
This commit is contained in:
Your Name
2026-07-03 04:14:18 +08:00
parent aeb436ba89
commit 8e9b7fa101
5 changed files with 31 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -22,7 +22,7 @@
| P0-1 | BLOCKED_HOST_WINDOWS | 全主機 reboot auto-detection / auto-trigger / 10 分鐘恢復 SLO | 2026-07-03 03:55 production scorecard：`active_blocker_count=8`、`readiness_percent=67`、`can_claim_all_services_recovered_within_target=false`；`observed_host_count=7`、`missing_host_count=0`、`unreachable_host_count=1`、111 unreachable、99 uptime unknown。 | 先收斂 99 / Windows99 / VMware 與 111：恢復 no-secret management channel 或 console verify stdout，讀回 VMX / VM power / host uptime，再 rerun host probe + reboot-event detector；不得 reboot、不得 VM power change、不得讀 Windows 密碼。 |
| P0-2 | BLOCKED_PUBLIC_UX | Deploy / reboot 期間 public 502 維護頁與外部 fallback | 多次 deploy marker 切換時 SLO/API route 曾短暫 HTTP 502 且沒有 maintenance body / fallback header；目前 03:55 public maintenance runtime readback 為 ready、raw 5xx count `0`，但 cutover-time 502 未被 drain / watch 消除。 | 實作 source-controlled deploy drain / maintenance fallback verifier / public route watch；完成條件是 marker-time probe 不再看到 raw 502，或看到明確 maintenance fallback header/body。 |
| P0-3 | PARTIAL_GREEN_SOURCE_RUNTIME | 所有產品 / 網站版本與資料最新性 | Gitea `main=89d4d6112`；Production SLO readback 對齊 deploy marker `89d4d6112 chore(cd): deploy 17ba08c [skip ci]`；Stock freshness `status=ok`、`latest_trading_date=2026-07-02`、blockers `[]`。AWOOOI health HTTP 200 但整體 `degraded`，SignOz / local Ollama 仍需列為 runtime degraded evidence。 | 將 source SHA / deploy marker / runtime endpoint / public route watch 固定進 scorecard；完成條件是每個 public product 都有 source、deploy、runtime、freshness 四層 readback，且 degraded components 有 owner lane。 |
| P0-4 | BLOCKED_WINDOWS99_AUTOSTART | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | Scorecard：`windows99_update_no_auto_reboot_ready=true`、`windows99_vmware_verify_ready=false`、VMX missing alias `111`、powered off aliases `111/112/120/121/188`、WinRM unavailable、SSH BatchMode permission denied、RDP / Hyper-V console channel reachable。 | 只用 no-secret console / management collector 取得 `windows99-vmware-autostart.ps1 -Mode Verify` 輸出；完成條件是 VMX config ready、guest power ready、99 uptime known、all required host reachable。 |
| P0-4 | BLOCKED_WINDOWS99_AUTOSTART | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | Scorecard：`windows99_update_no_auto_reboot_ready=true`、`windows99_vmware_verify_ready=false`、VMX missing alias `111`、powered off aliases `111/112/120/121/188`、WinRM unavailable、SSH BatchMode permission denied、RDP / Hyper-V console channel reachable。 | 只用 no-secret console / management collector 取得 `windows99-vmware-autostart.ps1 -Mode Verify` stdout，先以 `validate-windows99-console-verify-artifact.py` 轉成 normalized artifact，再重跑 SLO scorecard；完成條件是 VMX config ready、guest power ready、99 uptime known、all required host reachable。 |
| P0-5 | PARTIAL_GREEN_BACKUP_MONITORING | Gitea / 主機 / DB / 網站 / 服務 / 套件 / 工具 / log 備份監控告警 | Gitea repo bundle readback ready：expected `12`、rows `12`、missing `0`、failed `0`、sample restore dry-run ok；backup core green。這只證明 repo bundle / core backup，不等於 Gitea full dump、DB/settings/issues/packages/LFS、所有工具與 log 全量備份監控完成。 | 補齊 backup health textfile / Prometheus / Telegram receipt matrix：每個 backup scope 必須有 fresh、failed、age、restore-drill、alert-receipt；完成條件是沒有「不知道備份有沒有跑」的盲點。 |
| P0-6 | SOURCE_READY_ALERT_RECEIPT_BLOCKED | 主機關機 / 重啟 / SLO miss / backup failure Telegram 告警 | Source 已有 reboot / backup alert rules 與 per-blocker projection；scorecard `telegram_active_blocker_alert_required_count=8`。尚未完成 shutdown / reboot / backup alert 的脫敏 Telegram delivery receipt 全矩陣。 | 補 alert receipt readback：host down、host up、SLO miss、Windows99 blocker、backup stale/failed、deploy 502、freshness stale；完成條件是每類告警都有 sent / received / dedup / escalation evidence。 |
| P0-7 | SOURCE_READY_SLA_AUTOMATION | 固定排查順序、ETA / wait reason、自動化判斷與修復 | Scorecard 已固定 `current_phase=host_boot_detection_blocked`、`eta_or_wait_reason=reboot_event_readback_missing_eta_unavailable`、`fixed_triage_order`；但 10 分鐘內自動恢復仍未達標。 | 把每個 blocker 的 next_safe_action、post_verifier、forbidden_actions 接到自動 work item / Telegram / scorecard；完成條件是重啟後自動判斷、主動告警、主動 rerun verifier，不再人工臨場猜流程。 |