fix(reboot): use host111 as windows99 vm target
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 4m18s
CD Pipeline / build-and-deploy (push) Successful in 5m7s
CD Pipeline / post-deploy-checks (push) Successful in 1m53s

This commit is contained in:
Your Name
2026-07-03 16:42:14 +08:00
parent 81304e354a
commit c2dd14acba
13 changed files with 56 additions and 40 deletions

View File

@@ -22,7 +22,7 @@
| P0-1 | BLOCKED_REBOOT_EVENT_AND_HOST_REACHABILITY | 全主機 reboot auto-detection / auto-trigger / 10 分鐘恢復 SLO | 2026-07-03 15:49 verify-only artifact `/tmp/awoooi-reboot-event-verify-20260703-154907``reboot_detected=false``all_required_hosts_observed=false``all_required_hosts_in_reboot_window=false``recovery_deadline_status=target_window_elapsed``unreachable_hosts=["111"]`、99 reachable 但 `uptime_seconds=unknown`、188 reachable 但 `systemd_state=degraded` / `startup_active=failed`。production SLO 仍為 `blocked_reboot_auto_recovery_slo_not_ready`,不能宣稱 10 分鐘自動恢復。 | 固定先 rerun host probe + reboot-event detector verify-only並收斂 111 reachability、99 uptime / VMware readback、188 degraded/startup failed不得 reboot、不得 VM power change、不得讀 Windows 密碼,不得用 ping-only 或舊 boot evidence 當完成證據。 |
| P0-2 | DEPLOYED_RUNTIME_EDGE_DRIFT_BLOCKED | Deploy / reboot 期間 public 502 維護頁與外部 fallback | 2026-07-03 15:35 source fix 已部署讀回production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 保留 `edge_public_maintenance_live_config_drift``runtime_metric_active_blocker_count=8`,且 route fallback ready 不再清掉 edge Nginx drift。110 SLO artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-151219` 與 textfile 證明 edge 未 ready`awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"} 0`、active blocker `edge_public_maintenance_live_config_drift`。 | 後續才進 privileged edge apply 或 L1 cloud/CDN fallback decision完成條件是 edge fallback live config / maintenance file / `nginx -t` / reload / public header verifier 或外部 L1 fallback 全綠。不得讀密碼、不得用 app / Docker / host restart 掩蓋 edge fallback drift。 |
| P0-3 | PARTIAL_GREEN_DEPLOYED_SOURCE | 所有產品 / 網站版本與資料最新性 | Gitea `main=bc8d3af3f chore(cd): deploy dacdd90 [skip ci]`CD `#4558` log 顯示 API/Web build/push wrapper 全部 `rc=0`、deploy marker `dacdd90`、production deploy readback matched。Production SLO route 已讀回最新 API 行為:`runtime_metric_runtime_readback_added_blockers=["windows99_vmware_autostart_config_not_ready","windows99_update_no_auto_reboot_policy_not_ready"]``windows99_update_no_auto_reboot_ready=false`。Stock freshness 仍為 `status=ok``latest_trading_date=2026-07-02`、blockers `[]`。 | 版本層已補到 AWOOOI production完成整體 P0 仍需收斂 Windows99 VMX / service / policy / power 與 host reboot-event blockers。後續每個 public product 仍要維持 source SHA、deploy marker、runtime endpoint、public route watch、freshness 四層一致。 |
| P0-4 | BLOCKED_WINDOWS99_TASK_NOT_TRIGGERED | 192.168.0.99 VMware 自動啟動與 VM guest 110 / 188 / 120 / 121 / 112 | 2026-07-03 15:57 no-secret via-110 collector artifact `/tmp/awoooi-windows99-autostart-detail-20260703-155710`VMX `110 / 188 / 120 / 121 / 112` 全存在VMware services okWindows Update no-auto-reboot readyscheduled task `AWOOOI-Start-VMware-VMs` present/enabled/Ready`VMWARE_AUTOSTART_TASK_DETAIL last_run_time=1999-11-30T00:00:00 last_task_result=267011` (`0x41303`) 且五台 `VM_POWER running=0`。 | 將 `windows99_vmware_autostart_task_never_ran` 納入 active blocker / metrics / Telegram routing執行已建立的 autostart check-mode package再 rerun no-secret collector + SLO scorecard。不得從 scorecard 直接 power on VM、重啟 Windows host / VMware service、改 scheduled task / registry 或讀 Windows 密碼。 |
| P0-4 | BLOCKED_WINDOWS99_TASK_NOT_TRIGGERED | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | 2026-07-03 16:33 production readback 已證明 source/API/textfile 都會輸出 `windows99_vmware_autostart_task_never_ran`,但同時發現舊 source-of-truth 把 Windows99 default VM target 寫成 `110 / 188 / 120 / 121 / 112`,與本輪事故需求及 host probe `111 unreachable` 不一致。最新 source 修正為 `111 / 188 / 120 / 121 / 112`110 回到 P0 host / control-plane truth不再當 99 VMware guest default。 | 部署修正後重新跑 no-secret via-110 collector + SLO scorecard確認 `required_vm_aliases``powered_off_aliases`、check-mode package、validator 與 API readback 都以 111 為 Windows99 VM target。仍不得從 scorecard 直接 power on VM、重啟 Windows host / VMware service、改 scheduled task / registry 或讀 Windows 密碼。 |
| P0-5 | RUNTIME_READY_BACKUP_RECEIPT_GAP | Gitea / 主機 / DB / 網站 / 服務 / 套件 / 工具 / log 備份監控告警 | Gitea repo bundle readback readyexpected `12`、rows `12`、missing `0`、failed `0`、sample restore dry-run okbackup core green。2026-07-03 source / runtime 已部署 `awoooi_backup_alert_receipt_*` 指標與 Prometheus rules110 exporter 讀回 88 個 stage requirement、188 讀回 12 個 stage requirement`BackupAlertReceiptMetricMissing*` inactive`BackupAlertReceiptStageMissing` 已修成每 `host / receipt_channel` 聚合 pending110 一條、188 一條。 | 補 `/backup/alert-receipts/*.last_success` 脫敏 marker下一層仍要補 Gitea full dump、DB/settings/issues/packages/LFS、所有工具與 log 全量備份監控。 |
| P0-6 | RUNTIME_READY_ALERT_RECEIPT_GAP | 主機關機 / 重啟 / SLO miss / backup failure Telegram 告警 | Reboot per-blocker alert 與 backup receipt alert rules 已 deploy/readbackbackup receipt 缺段不再產生 100 條 stage 噪音,現在聚合成 110 / 188 兩條 host-level pending。scorecard 目前有 10 個 reboot active blockers其中 Windows99 config / policy blocker 已可供 per-blocker alert routing 使用;尚未完成 shutdown / reboot / backup alert 的 production 脫敏 delivery receipt 全矩陣。 | 補 alert receipt readbackhost down、host up、SLO miss、Windows99 blocker、backup stale/failed、deploy 502、freshness stale完成條件是每類告警都有 sent / received / dedup / escalation evidence。 |
| P0-7 | SOURCE_READY_SLA_AUTOMATION | 固定排查順序、ETA / wait reason、自動化判斷與修復 | Scorecard 已固定 `current_phase``eta_or_wait_reason``primary_blocker`、fixed triage order 與 action matrix2026-07-03 新增 `windows99_vmware_autostart_task_never_ran`,其 next action 固定為 `run_windows99_vmware_autostart_check_mode_package_for_powered_off_aliases_then_rerun_no_secret_collector_no_vm_power_change`,不再退回泛用「再收一次 verify」。 | 把每個 blocker 的 next_safe_action、post_verifier、forbidden_actions 接到自動 work item / Telegram / scorecard完成條件是重啟後自動判斷、主動告警、主動 rerun verifier不再人工臨場猜流程。 |