fix(reboot): expose windows99 management channel readback
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 1s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Failing after 41s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped

This commit is contained in:
Your Name
2026-07-02 15:19:09 +08:00
parent 25148d0752
commit aaa617f00f
14 changed files with 983 additions and 195 deletions

View File

@@ -57,15 +57,15 @@
| 順序 | ID | 優先序 | 使用者插入要求 | 正規化工作項 | 目前狀態 | 下一個可驗證動作 |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | CIR-P0-RBT-001 | P0 | 「主機重啟後 10 分鐘內全部恢復,且要自動判斷所有主機被重啟」 | 建立 99/110/111/112/120/121/188 reboot event detector + 10 分鐘 SLO scorecard + fixed triage order | 部分已有 `reboot_recovery_slo_alerts`、scorecard、textfile仍需要 fresh all-host reboot/drill 證明 | 產生最新 reboot SLO scorecard readback若缺 fresh event`awaiting_next_reboot_or_approved_drill`不可宣稱 10 分鐘 SLA 已證明 |
| 2 | CIR-P0-RBT-002 | P0 | 「沒有偵測到主機重啟」 | 修正 host reboot/shutdown/up detectionboot_id / uptime / node exporter / Windows exporter / VMware VM power state 都要進同一事件 | Source verifier 已補scorecard 可解析 Windows99 VMware readback`windows99_verify_collection` 已補 no-secret collection packetlive 99 verify output 仍缺 | 收集 `windows99-vmware-autostart.ps1 -Mode Verify` no-secret output 後 rerun scorecard缺 99 時不得把 110/120/121/188 green 當全主機 green |
| 3 | CIR-P0-RBT-003 | P0 | 「192.168.0.99 VMWare 要自動啟動,裡面 111/188/120/121/112 也自動啟動」 | Windows 99 VMware host autostart + guest VM autostart contractVM host 111/188/120/121/112 開機順序與 readback | Source verifier / parser / API readback / collection packet 已完成snapshot active blocker=`windows99_vmware_autostart_readback_missing` | 從 99 取得 no-secret Verify output確認 `VMRUN_PRESENT`、scheduled task、VMware services、VM power、VMX present 全綠 |
| 4 | CIR-P0-RBT-004 | P0 | 「192.168.0.99 不可因 Windows Update 無預警重開」 | Windows Update reboot policyactive hours / no auto-restart / maintenance window / update notification audit | Source verifier 已補 `WINDOWS_UPDATE_POLICY``WINDOWS_UPDATE_NO_AUTO_REBOOT_READY`collection packet 已列 forbidden actionslive 99 policy readback 仍缺 | 從 99 取得 Verify output若 policy 不綠,再走 controlled apply禁止要求或記錄 Windows 密碼 |
| 1 | CIR-P0-RBT-001 | P0 | 「主機重啟後 10 分鐘內全部恢復,且要自動判斷所有主機被重啟」 | 建立 99/110/111/112/120/121/188 reboot event detector + 10 分鐘 SLO scorecard + fixed triage order | 2026-07-02 15:08 live scorecard 已更新readiness `43%`、active blockers `11``windows99_verify_collection``windows99_management_channel` 已進 API / scorecard仍缺 fresh all-host 10 分鐘證明111 不可達99 uptime / VMware verifier 未閉環 | 優先收斂 99 no-secret management channel / verifier readback 與 111 reachability不可宣稱 10 分鐘 SLA 已證明 |
| 2 | CIR-P0-RBT-002 | P0 | 「沒有偵測到主機重啟」 | 修正 host reboot/shutdown/up detectionboot_id / uptime / node exporter / Windows exporter / VMware VM power state 都要進同一事件 | Scorecard 已接 collection packet + management probe99 host reachable 但 uptime unknown111 unreachablestale hosts 仍存在 | 讓 99 verifier / Windows exporter 或等效 no-secret readback 進入 host boot event並補 111 reachability 證據 |
| 3 | CIR-P0-RBT-003 | P0 | 「192.168.0.99 VMWare 要自動啟動,裡面 111/188/120/121/112 也自動啟動」 | Windows 99 VMware host autostart + guest VM autostart contractVM host 111/188/120/121/112 開機順序與 readback | Source verifier / parser / API readback / collection packet 已完成;management probe 讀回 `host_reachable=true`、RDP open、SSH BatchMode `permission_denied`、WinRM timeoutsnapshot active blockers=`windows99_remote_execution_channel_unavailable``windows99_vmware_autostart_readback_missing` | 恢復 no-secret management channel 或收集 local console Verify output確認 `VMRUN_PRESENT`、scheduled task、VMware services、VM power、VMX present 全綠 |
| 4 | CIR-P0-RBT-004 | P0 | 「192.168.0.99 不可因 Windows Update 無預警重開」 | Windows Update reboot policyactive hours / no auto-restart / maintenance window / update notification audit | Source verifier 已補 `WINDOWS_UPDATE_POLICY``WINDOWS_UPDATE_NO_AUTO_REBOOT_READY`collection packet 已列 forbidden actions99 management channel 尚不能收 policy readback | 取得 Verify output若 policy 不綠,再走 controlled apply禁止要求或記錄 Windows 密碼 |
| 5 | CIR-P0-RBT-005 | P0 | 「網站重啟後 502 嚴重影響體驗,要維護頁,外部雲端或專業做法」 | Public maintenance fallbackNginx / edge / external static maintenance page / status page / fail-open UX避免 502 直出 | 尚未完整落地;目前是需求缺口 | 產生 `public_maintenance_fallback` decision recordDNS/edge/外部雲端/本地 Nginx fallback 風險比較,先做不切流量的 check-mode |
| 6 | CIR-P0-RBT-006 | P0 | 「所有主機關機立刻 Telegram 告警,重啟後也要告警,其他告警一併完整思考」 | Down / shutdown suspected / reboot detected / reboot recovered / SLO missed / backup failed / freshness stale / CPU pressure / Gitea queue 告警矩陣 | 部分已有 Alertmanager rule 與 Telegram receipt 補強;仍缺完整 shutdown/up E2E receipt | 建立 Telegram alert matrix + receipt verifier逐項讀回 Alertmanager active/resolved 與 outbound receipt不送測試 secret |
| 7 | CIR-P0-RBT-007 | P0 | 「所有備份包含主機、DB、網站、服務、套件、工具、日誌都沒有監控告警」 | Backup observability coveragebackup job inventory、last success、freshness、offsite、restore drill、Telegram receipt | 部分已有 backup health exporter / alert rules全域 coverage 與 restore drill 未全綠 | 建立 backup coverage matrixhost / DB / website / service config / package list / tool scripts / logs每列有 metric、alert、last_success、restore_verifier |
| 8 | CIR-P0-RBT-008 | P0 | 「每次重啟排查都不一樣,也不知道多久恢復,不符合 SLA」 | 固定化 reboot runbookfixed triage order、ETA、active blocker、remaining seconds、owner lane、next command | 已補 scorecard source / snapshot / API 固定欄位:`current_phase``eta_or_wait_reason``primary_blocker``next_safe_action`;仍需 production deploy readback 與下一次 fresh reboot/drill 證明 | 推送後讀回 `/api/v1/agents/reboot-auto-recovery-slo-scorecard`,確認固定欄位在 production 可見;再接 99/VMwarefresh reboot/drill |
| 9 | CIR-P0-RBT-009 | P0 | 「所有產品、網站都要是最新版本;版本和數據是否最新要驗證」 | Product freshness/version matrixsource commit、deploy marker、runtime image、public health、data freshness、latest source availability | AWOOOI / StockPlatform 部分已在做;全產品未統一 | 建立全產品 readback 表product、canonical repo、main SHA、deploy marker、public URL、data freshness、blocked reason |
| 8 | CIR-P0-RBT-008 | P0 | 「每次重啟排查都不一樣,也不知道多久恢復,不符合 SLA」 | 固定化 reboot runbookfixed triage order、ETA、active blocker、remaining seconds、owner lane、next command | Scorecard/API contract 已固定 `next_safe_action=restore_windows99_no_secret_management_channel_or_collect_local_console_verify_readback_then_rerun_reboot_scorecard_no_reboot`;尚待 deploy readback | 推送後讀回 `/api/v1/agents/reboot-auto-recovery-slo-scorecard`,確認 production 顯示 99 management-channel blocker 與 readiness `43%` |
| 9 | CIR-P0-RBT-009 | P0 | 「所有產品、網站都要是最新版本;版本和數據是否最新要驗證」 | Product freshness/version matrixsource commit、deploy marker、runtime image、public health、data freshness、latest source availability | 本輪 StockPlatform public freshness / ingestion 讀回 `ok`latest trading date `2026-07-01`;全產品版本矩陣仍未統一 | 建立全產品 readback 表product、canonical repo、main SHA、deploy marker、public URL、data freshness、blocked reason |
| 10 | CIR-P0-GIT-001 | P0 | 「Gitea 儲存庫都不見了Gitea 沒完整備份嗎?」 | Gitea repository identity + backup proof + restore drill不能只看 UI visible要比對 SSH heads、repo path、bundle backup、restore sample | 已有 9 expected repos OK / backup health missing=0 的 handoff仍需 restore drill 證明 | 補 Gitea repo bundle backup readback + sample restore dry-run verifier禁止刪 repo / 改 visibility |
| 11 | CIR-P0-CPU-001 | P0 | 「110 / 188 CPU 負載持續過高,為什麼沒監控告警、沒主動修復」 | Sustained CPU pressure automationAlertmanager → controller → evidence → service playbook → verifier → KM writeback | 110 已有 `Host110SustainedModeratePressure`、Gitea playbook、Stock/Postgres evidence188 仍需同級 controller/alerts readback | 下一步接 `postgres_hot_query_or_backup_export_playbook`;並補 188 equivalent readback不以單次下降結案 |
| 12 | CIR-P0-CPU-002 | P0 | 「噪音會影響真問題,要整合一起做」 | Alert noise / real issue correlationbackup aggregate noise、CPU pressure、Gitea queue、Stock freshness 要分清主因與次因 | 部分已在 SOP 註記;仍需統一 correlation scorecard | 建立 incident correlation readbackprimary_blocker、secondary_noise、ignored_noise_reason、evidence_ref |