feat(recovery): close host 112 guest readiness
This commit is contained in:
ogt
@@ -54675,3 +54675,27 @@ production browser smoke:
|
||||
|
||||
**下一步**：
|
||||
- commit / push 到 Gitea main；deploy 後讀回 production locator endpoint 與 priority readback。接續 P0 是提供 / 找到 111 verified VMX source 或 backup path，再產生 scoped dry-run，不直接 relink 或 power on。
|
||||
|
||||
## 2026-07-11 - P0-008 112 Kali 真正開機完成與自動恢復閉環
|
||||
|
||||
**事故判讀**：
|
||||
- Windows99 的 SYSTEM scheduled task 以 headless 方式啟動 112 VMX；Administrator VMware Workstation session 因 principal/session 隔離顯示 `Powered off`，但 VMX process、Ping、SSH 與 guest boot identity 都證明 VM 正在執行。
|
||||
- 112 Guest 當時預設為 `multi-user.target`，LightDM 未啟動；`wazuh-indexer.service` 在開機競爭期間觸及 3 分鐘 timeout。舊 SOP 只看 VM power、Ping/SSH/node-exporter，錯誤把「可達」視為「完整開機」。
|
||||
|
||||
**受控修復**：
|
||||
- live 恢復 `graphical.target`、LightDM/Xorg、open-vm-tools 與 Wazuh Indexer，systemd 回 `running`。
|
||||
- 新增 `host112-guest-readiness.sh`、systemd service/timer 與 installer；timer 開機 20 秒後啟動、每 60 秒檢查，Index retry 具 210 秒 boot delay 與 600 秒 cooldown。
|
||||
- 建立 110 到 112 的來源限制 forced-command 公鑰讀回與 exact-command sudo executor；任何 VM power、host reboot、firewall、DB 或 secret 操作都不在 allowlist。
|
||||
- Host probe / scorecard / exporter 新增 Console、Wazuh、recovery timer 與 guest-ready gates；Prometheus 新增 `Host112GuestReadbackMissing` 與 `Host112GuestNotReady`。
|
||||
- Windows99 VMware verifier新增 VMX owner/session readback，112 明確輸出 `runtime_running_gui_session_isolated`，避免依錯誤 VMware UI 再次 power on。
|
||||
|
||||
**驗證結果**：
|
||||
- 112：`systemd=running`、`console_ready=1`、`services_ready=1`、`recovery_timer_ready=1`、`guest_ready=1`。
|
||||
- 110 artifact：`/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260711-194901`；`host112_* blocker=0`。
|
||||
- Prometheus：五項 112 gauges `5/5`；兩條專屬規則 health `ok`、state `inactive`；source/canonical/container SHA 一致，drift guard `repaired=0`。
|
||||
- Windows99：112 VMX PID `22288`、session `0`、owner `NT AUTHORITY\\SYSTEM`；VMware verify ready。
|
||||
- 測試：`pytest -q scripts/reboot-recovery/tests` 為 `142 passed`；JSON/YAML parse 與 bash syntax 通過。
|
||||
- Receipt：`docs/operations/host112-guest-recovery-20260711.snapshot.json`；runtime rollback installer 位於 `/usr/local/sbin/awoooi-install-host112-guest-recovery`。
|
||||
|
||||
**剩餘驗收**：
|
||||
- 112 runtime scope 已恢復並驗證；全主機 program 仍保留歷史 10 分鐘逾時，不以事後綠燈覆蓋。下一次真實全主機 reboot 必須在同一 600 秒 window 重新通過 VMX、112 Console/Wazuh/timer、七主機、服務、資料、備份與 public fallback。
|
||||
|
||||
@@ -461,24 +461,24 @@
|
||||
"order": 8,
|
||||
"priority": "P0",
|
||||
"status": "in_progress",
|
||||
"title": "Windows 99 VMware autostart, VMX, lock and update policy",
|
||||
"title": "Windows 99 VMware autostart, VMX, 112 guest readiness, lock and update policy",
|
||||
"owner_lane": "windows99_vmware",
|
||||
"dependencies": ["AG99-P0-003", "AG99-P0-005"],
|
||||
"scope_refs": ["host:99", "host:110", "host:111", "host:112", "host:120", "host:121", "host:188"],
|
||||
"problem": "The drill showed an 18-minute-45-second delay between Windows boot and the VMware autostart task, a principal visibility mismatch for the Kali VM and two later VIX SoftReboot actions without an attributable trace. Current VM power is green, but the next startup must prove timing and every reset/stop/suspend must be audited.",
|
||||
"problem": "The drill showed an 18-minute-45-second delay between Windows boot and the VMware autostart task, a principal visibility mismatch for the Kali VM and two later VIX SoftReboot actions without an attributable trace. The 2026-07-11 follow-up proved that VMX, Ping, SSH and node-exporter were insufficient for host 112: Kali booted to multi-user.target without LightDM, Wazuh Indexer timed out during boot, and the Administrator VMware UI falsely displayed Powered off while a SYSTEM-owned VMX process was running.",
|
||||
"trigger": "Windows startup, VMware exclusive-lock error, guest power mismatch, VMX drift or Windows Update policy drift.",
|
||||
"ai_role": "Windows and VMware control agent",
|
||||
"action_mode": "check_mode_then_controlled_apply",
|
||||
"evidence_sources": ["VMX inventory", "vmrun list", "VMware services", "scheduled tasks", "lock owners", "Windows Update policy"],
|
||||
"controlled_actions": ["repair stale lock after owner proof", "configure autostart order", "verify guest tools and network", "apply reversible update policy"],
|
||||
"evidence_sources": ["VMX inventory and process principal", "vmrun list", "guest boot identity and uptime", "112 forced-command readiness", "graphical target and LightDM", "VMware Tools", "Wazuh services", "scheduled tasks", "lock owners", "Windows Update policy"],
|
||||
"controlled_actions": ["repair stale lock after owner proof", "configure autostart order", "verify guest tools and network", "start host 112 graphical target, LightDM and VMware Tools", "retry Wazuh Indexer after boot with cooldown", "apply reversible update policy"],
|
||||
"prohibited_actions": ["delete unknown lock", "guess VMX path", "power off VM", "reset stop or suspend a VM without trace and verifier", "reboot Windows without break-glass"],
|
||||
"rollback": "Restore previous task, service and policy exports; keep guest power unchanged during check mode.",
|
||||
"verifier": "All required VMX paths exist, guests report running and reachable, tasks survive reboot, update policy drift is zero.",
|
||||
"verifier": "All required VMX paths exist; guests report running and reachable; host 112 returns systemd running, graphical target, LightDM, Xorg, VMware Tools, Wazuh stack and recovery timer green through a forced-command readback; tasks survive reboot; update policy drift is zero.",
|
||||
"callback": "Update reboot SLO VMware and Windows policy gates.",
|
||||
"acceptance": "99 startup automatically launches VMware guests 110/188/112/120/121 in order, physical host 111 is verified separately, no exclusive-lock failure occurs and Windows Update cannot surprise-reboot the host.",
|
||||
"implementation_progress": "The controlled apply receipt windows99-vmware-autostart-apply-20260711-173505.json completed DryRun, backup, Apply and independent Verify. VMX process count remained 5, required guests 110/188/120/121/112 are running, the generated startup script has a process-aware duplicate-start guard, the task and Windows Update policy are green, and no reboot, VM power change or secret read occurred. VMware logs prove the 120/121 secondary reboots were VIX SoftReboot actions outside Agent99; actor attribution was unavailable because process auditing was not enabled.",
|
||||
"evidence_refs": ["C:\\Wooo\\Agent99\\evidence\\windows99-vmware-autostart-apply-20260711-173505.json", "C:\\Wooo\\Agent99\\deploy\\vmware-autostart-backup-20260711-173505", "/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260711-172623/windows99-vmware-verify.txt"],
|
||||
"next_action": "Keep guest power unchanged, add trace-correlated VMware control auditing, and on the next approved drill verify that the startup task runs promptly after Windows boot and all five guests plus physical host 111 recover inside the same 600-second window."
|
||||
"acceptance": "99 startup automatically launches VMware guests 110/188/112/120/121 in order; physical host 111 is verified separately; 112 must pass Console-ready and Wazuh service gates rather than only Ping or SSH; no exclusive-lock failure occurs; Windows Update cannot surprise-reboot the host.",
|
||||
"implementation_progress": "The controlled VMware apply remains green with five running VMX processes. The 112 follow-up restored graphical.target, LightDM and Wazuh Indexer, deployed a bounded guest recovery timer, restricted 110 forced-command readback, exact-command sudo executor and durable rollback installer. The 110 scorecard now reads systemd running plus Console/Wazuh/timer/guest 1/1/1/1, all five host112 Prometheus gauges are 1, host112 blockers are zero, both dedicated alert rules are healthy/inactive, and Windows99 records the SYSTEM session VMX as runtime_running_gui_session_isolated. Reboot-recovery tests pass 142/142.",
|
||||
"evidence_refs": ["C:\\Wooo\\Agent99\\evidence\\windows99-vmware-autostart-apply-20260711-173505.json", "C:\\Wooo\\Agent99\\deploy\\vmware-autostart-backup-20260711-173505", "/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260711-194901", "/var/lib/awoooi-host112-recovery/backups/host112-guest-recovery-20260711T195449+0800-43505", "docs/operations/host112-guest-recovery-20260711.snapshot.json", "reboot-recovery suite 142 passed"],
|
||||
"next_action": "Keep guest power unchanged, route any host112 state change to Agent99 controlled apply and lifecycle callback, then prove the same Console/Wazuh/timer gates inside the next real all-host 600-second drill."
|
||||
},
|
||||
{
|
||||
"id": "AG99-P0-009",
|
||||
@@ -658,9 +658,13 @@
|
||||
"dependencies": ["AG99-P0-006"],
|
||||
"scope_refs": ["surface:Telegram", "all:incidents"],
|
||||
"action_mode": "controlled_apply",
|
||||
"problem": "Current Agent99 Telegram output exposes verifier field names, Windows paths, truncated summaries and duplicated screenshot text. It does not give operators a concise impact statement, lifecycle state, actual AI action, verified result, duration or next update, and duplicate same-state messages create alert fatigue.",
|
||||
"controlled_actions": ["render detected, investigating, remediating and recovered lifecycle cards", "use Traditional Chinese operator summary", "attach real trend chart, topology or screenshot only when it adds evidence", "edit or thread the same incident message", "suppress duplicate fingerprints until state changes"],
|
||||
"prohibited_actions": ["send raw verifier dumps", "repeat the same text as an image", "expose internal paths or secrets", "declare resolved from send success", "send periodic same-state noise"],
|
||||
"verifier": "Message snapshot tests, duplicate suppression, state-change delivery and evidence-link checks.",
|
||||
"acceptance": "Every card explains impact, AI action, verified result and next step in Traditional Chinese; charts or screenshots are attached when they improve understanding.",
|
||||
"next_action": "Replace raw technical completion messages with one lifecycle card per incident state change."
|
||||
"acceptance": "Every card explains severity, affected service or host, user impact, lifecycle state, AI diagnosis, controlled action, verifier result, duration, recurrence count and next update in Traditional Chinese. Charts or real screenshots are attached only when they improve understanding; one fingerprint produces one lifecycle thread and no same-state spam.",
|
||||
"implementation_progress": "The 2026-07-11 Agent99 incident-card screenshot was captured as a failing UX fixture and added to the P1 acceptance contract; runtime renderer replacement remains pending behind the current host 112 P0 closure.",
|
||||
"next_action": "Replace the raw automation-result renderer with detected, progress and recovered card templates; add fingerprint edit/thread dedup, redaction, chart-or-screenshot evidence rules and Telegram delivery snapshot tests."
|
||||
},
|
||||
{
|
||||
"id": "AG99-P1-004",
|
||||
|
||||
140
docs/operations/host112-guest-recovery-20260711.snapshot.json
Normal file
140
docs/operations/host112-guest-recovery-20260711.snapshot.json
Normal file
@@ -0,0 +1,140 @@
|
||||
{
|
||||
"schema_version": "host112_guest_recovery_receipt_v1",
|
||||
"generated_at": "2026-07-11T19:52:31+08:00",
|
||||
"work_item_id": "AG99-P0-008",
|
||||
"trace_id": "host112-guest-recovery-20260711T193137+0800-35698",
|
||||
"incident_fingerprint": "host112_vmware_ui_false_powered_off_guest_headless_wazuh_boot_timeout",
|
||||
"asset": {
|
||||
"canonical_id": "host:112",
|
||||
"address": "192.168.0.112",
|
||||
"guest_role": "Kali and Wazuh security node",
|
||||
"vm_owner_host": "host:99",
|
||||
"vmx_alias": "112"
|
||||
},
|
||||
"sensor_receipt": {
|
||||
"vmware_ui_observation": "Powered off",
|
||||
"vmx_process_present": true,
|
||||
"vmx_process_id": 22288,
|
||||
"vmx_process_owner": "NT AUTHORITY\\SYSTEM",
|
||||
"vmx_process_session_id": 0,
|
||||
"guest_boot_id": "f265ad96-a262-4285-9836-d2dab7eefb97",
|
||||
"guest_network_reachable": true,
|
||||
"guest_ssh_reachable": true,
|
||||
"pre_recovery_default_target": "multi-user.target",
|
||||
"pre_recovery_lightdm": "inactive",
|
||||
"pre_recovery_wazuh_indexer": "failed_timeout",
|
||||
"wazuh_indexer_boot_timeout": "3min",
|
||||
"wazuh_indexer_boot_memory_peak": "1.2G"
|
||||
},
|
||||
"normalized_diagnosis": {
|
||||
"vm_power_state": "running",
|
||||
"vmware_ui_state": "untrusted_cross_principal_false_powered_off",
|
||||
"guest_boot_state": "reachable_but_not_console_or_service_ready",
|
||||
"root_causes": [
|
||||
"SYSTEM-owned headless VMX is isolated from the Administrator VMware UI session",
|
||||
"Kali defaulted to multi-user.target and LightDM was inactive",
|
||||
"Wazuh Indexer timed out during boot contention and recovered on a delayed retry"
|
||||
]
|
||||
},
|
||||
"source_diff": {
|
||||
"new_assets": [
|
||||
"scripts/reboot-recovery/host112-guest-readiness.sh",
|
||||
"scripts/reboot-recovery/awoooi-host112-guest-recovery.service",
|
||||
"scripts/reboot-recovery/awoooi-host112-guest-recovery.timer",
|
||||
"scripts/reboot-recovery/install-host112-guest-recovery.sh"
|
||||
],
|
||||
"updated_assets": [
|
||||
"scripts/reboot-recovery/windows99-vmware-autostart.ps1",
|
||||
"scripts/reboot-recovery/reboot-auto-recovery-host-probe.sh",
|
||||
"scripts/reboot-recovery/reboot-auto-recovery-slo-scorecard.py",
|
||||
"scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh",
|
||||
"ops/monitoring/alerts-unified.yml",
|
||||
"docs/runbooks/REBOOT-RECOVERY-SOP.md"
|
||||
],
|
||||
"source_sha256": {
|
||||
"host112_readiness": "dc874bf9defe655cf2cae7a678783a847b6653b4255ef5798903afbc974e3e2d",
|
||||
"host112_installer": "c0ea2d8c10aebef6079f4f73f8614cd9c3fb875903b55824f6b44acf23ab40db",
|
||||
"windows99_vmware_verifier": "437cc7766f06d9fd543cc472569b238149a79f2dff454dbb9ed704d9e3be7648",
|
||||
"host_probe": "31dce6f90dc89e2fd38bcde91049aa79711b3e5e11a1ae2d1053e27bbbc1f01f",
|
||||
"scorecard": "835abf7bf6cf587d40467351974f5eba2b84f731c78c2e6c7aaa254f0eaaf33d",
|
||||
"exporter": "85f16a3d190cdca866ad44f03c4c3e4decfbdf0d3af873c8e4af4d66780d5214",
|
||||
"alert_rules": "2f606acd9d35b3708f57f313fcfe1f75c7b274316de0b27f494636a132e8ba4d"
|
||||
}
|
||||
},
|
||||
"decision": {
|
||||
"risk": "medium",
|
||||
"controlled_apply_allowed": true,
|
||||
"candidate_action": "Install a bounded guest recovery timer, restricted forced-command readback, exact-command sudo executor, scorecard gates and dedicated alerts.",
|
||||
"forbidden_actions": [
|
||||
"host reboot",
|
||||
"VM power off, reset or duplicate power on",
|
||||
"firewall change",
|
||||
"database restore or write",
|
||||
"secret or private key value read"
|
||||
]
|
||||
},
|
||||
"check_receipt": {
|
||||
"staged_source_hashes_matched": true,
|
||||
"control_public_key_fingerprint_verified": true,
|
||||
"focused_contract_tests": "31 passed",
|
||||
"json_yaml_parse": "passed"
|
||||
},
|
||||
"execution_receipt": {
|
||||
"initial_recovery_run_id": "host112-guest-recovery-20260711T193137+0800-35698",
|
||||
"durable_installer_run_id": "host112-guest-recovery-20260711T195449+0800-43505",
|
||||
"backup_dirs": [
|
||||
"/var/lib/awoooi-host112-recovery/backups/host112-guest-recovery-20260711T193137+0800-35698",
|
||||
"/var/lib/awoooi-host112-recovery/backups/host112-guest-recovery-20260711T195449+0800-43505"
|
||||
],
|
||||
"durable_installer_path": "/usr/local/sbin/awoooi-install-host112-guest-recovery",
|
||||
"timer_enabled": true,
|
||||
"timer_active": true,
|
||||
"restricted_control_key_installed": true,
|
||||
"sudo_exact_command_installed": true,
|
||||
"host_reboot_performed": false,
|
||||
"vm_power_change_performed": false,
|
||||
"secret_read": false
|
||||
},
|
||||
"post_verifier": {
|
||||
"host112_guest_ready": true,
|
||||
"host112_console_ready": true,
|
||||
"host112_services_ready": true,
|
||||
"host112_recovery_timer_ready": true,
|
||||
"host112_systemd_state": "running",
|
||||
"host112_default_target": "graphical.target",
|
||||
"host112_lightdm": "active",
|
||||
"host112_vmware_tools": "active",
|
||||
"host112_wazuh_indexer": "active",
|
||||
"host112_wazuh_manager": "active",
|
||||
"host112_wazuh_dashboard": "active",
|
||||
"host112_filebeat": "active",
|
||||
"forced_command_from_110": "passed",
|
||||
"sudo_n_controlled_apply": "passed_no_action_needed",
|
||||
"scorecard_artifact": "/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260711-194901",
|
||||
"host112_active_blocker_count": 0,
|
||||
"prometheus_host112_metrics_green": "5/5",
|
||||
"prometheus_rule_health": "2/2 ok inactive",
|
||||
"prometheus_rule_sha_source_canonical_runtime_matched": true,
|
||||
"prometheus_rule_drift_guard_repaired": false,
|
||||
"windows99_vmware_verify_ready": true,
|
||||
"windows99_operator_power_truth": "trust_vmx_process_and_guest_readiness_not_interactive_vmware_ui",
|
||||
"reboot_recovery_tests": "142 passed"
|
||||
},
|
||||
"rollback": {
|
||||
"command": "sudo /usr/local/sbin/awoooi-install-host112-guest-recovery --rollback /var/lib/awoooi-host112-recovery/backups/host112-guest-recovery-20260711T195449+0800-43505",
|
||||
"tested": false,
|
||||
"reason_not_executed": "Post-verifier passed; destructive rollback rehearsal was not required on the recovered production guest."
|
||||
},
|
||||
"closure": {
|
||||
"host112_runtime_scope": "recovered_and_verified",
|
||||
"full_host_reboot_program": "in_progress_next_600_second_drill_required",
|
||||
"remaining_reboot_scorecard_blockers": [
|
||||
"all_required_hosts_not_in_10_minute_reboot_window",
|
||||
"fresh_all_host_reboot_event_missing",
|
||||
"host_boot_observation_older_than_target_window"
|
||||
],
|
||||
"telegram_receipt": "not_sent_healthy_state_no_duplicate_notification",
|
||||
"km_writeback": "docs/runbooks/REBOOT-RECOVERY-SOP.md",
|
||||
"work_item_writeback": "docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json"
|
||||
}
|
||||
}
|
||||
@@ -69,6 +69,16 @@ L0 source contract 是 `scripts/reboot-recovery/external-l0-recovery-supervisor.
|
||||
|
||||
Windows 99 VMware task 已經由 `scripts/reboot-recovery/windows99-vmware-autostart-controlled-apply.ps1` 正式受控套用：DryRun、Apply、Verify 同一 receipt；task XML、舊 startup script、Windows Update policy 與 service start modes 都有 rollback。Evidence `C:\Wooo\Agent99\evidence\windows99-vmware-autostart-apply-20260711-173505.json` 回報 pre/post VMX process 都是 `5`、running guest rows `5`、process-aware duplicate-start guard true、沒有 host reboot、沒有 VM power change、沒有 secret read。這才是 Agent99 與重啟 SOP 的 runtime 整合證據；仍不取代下一次 600 秒 E2E drill。
|
||||
|
||||
### 112 Kali 真正開機完成 Gate
|
||||
|
||||
112 一直存在於 Windows99 required VM inventory、host probe 與 reboot event required host set；本次事故暴露的缺陷是舊流程只用 VMX process、Ping、SSH port 或 node-exporter 判定「已開機」，沒有驗證 Kali 的圖形登入與安全服務。這會讓 112 實際以 `multi-user.target` 無 GUI 啟動、LightDM 未執行且 Wazuh Indexer 開機逾時時仍被錯誤判成綠燈。另一個獨立現象是 SYSTEM principal 以 `vmrun ... nogui` 啟動的 VMX 不會出現在 Administrator 的 VMware Workstation session，GUI 可能顯示 `Powered off`；在 VMX process 與 guest boot identity 已存在時，禁止因此再次按 Power on，避免 exclusive lock 與重複啟動。
|
||||
|
||||
112 的固定完成鏈改為：`VMX process -> guest boot_id/uptime -> SSH forced-command readback -> systemd running -> graphical.target -> LightDM/display-manager -> Xorg -> open-vm-tools -> Wazuh Indexer/Manager/Dashboard/Filebeat -> recovery timer -> Console-ready`。`scripts/reboot-recovery/host112-guest-readiness.sh` 提供 no-secret `--check` 與 allowlisted `--apply`；apply 只可補起 graphical target、LightDM、VMware Tools，以及在開機 210 秒後、600 秒 cooldown 內重試一次 Wazuh Indexer。`awoooi-host112-guest-recovery.timer` 於開機 20 秒後開始、每 60 秒檢查；不得 reboot host、reset/power cycle VM、修改 firewall、讀取 secret 或執行資料庫寫入。
|
||||
|
||||
110 到 112 使用來源限制、`restrict` 與 forced command 的既有公鑰，只能執行 `/usr/local/sbin/awoooi-host112-guest-readiness --check`，不能取得 shell 或 forwarding。Host probe 必須輸出 `console_ready=1`、`services_ready=1`、`recovery_timer_ready=1`、`guest_ready=1`；任何一項缺失都建立具名 `host112_*` blocker，阻止 full-host recovery 與 maintenance withdrawal。Prometheus 固定輸出 `awoooi_host112_guest_readback_present`、`awoooi_host112_console_ready`、`awoooi_host112_services_ready`、`awoooi_host112_recovery_timer_ready`、`awoooi_host112_guest_ready`，Agent99 收到狀態變更後執行受控修復並以同一 incident 回報 verifier 結果。
|
||||
|
||||
2026-07-11 runtime closure：installer runs `host112-guest-recovery-20260711T193137+0800-35698` 與 `host112-guest-recovery-20260711T195449+0800-43505` 已完成備份、受限公鑰、exact-command sudo、timer 與 durable rollback installer，未 reboot、未改 VM power、未讀 secret。110 artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260711-194901` 讀回 112 `systemd=running`、Console/Wazuh/timer/guest `1/1/1/1`、五項 Prometheus gauges `5/5`、`host112_* blocker=0`。兩條 112 告警規則 health `2/2 ok` 且 healthy state 為 inactive；source/canonical/container SHA 一致，drift guard `repaired=0`。Windows99 verifier 同輪讀到 `VM_PROCESS alias=112 process_id=22288 session_id=0 owner=NT_AUTHORITY\SYSTEM ui_truth=runtime_running_gui_session_isolated`，因此 operator power truth 固定信任 VMX process + guest readiness，不信任跨 principal 的互動式 VMware UI。完整 receipt 見 `docs/operations/host112-guest-recovery-20260711.snapshot.json`；reboot-recovery tests `142 passed`。下一個未關閉驗收只剩下一次真實全主機 600 秒 drill，不能用這次事後恢復覆蓋歷史逾時。
|
||||
|
||||
### 2026-07-10 P0 Agent99 Transport / Verified Remediation 覆蓋
|
||||
|
||||
99 的 Agent99 是全主機恢復控制節點。Windows 開機後由既有 Agent99 scheduled tasks 啟動；`Status` 讀到 boot identity 變更時，只能用 single-flight queue 觸發一次 `Recover`，並以 10 分鐘 cooldown 避免併發恢復。每次執行必須寫入 `bootState`、`recoveryTrigger`、`recoveryCoordinator` 與 `recoverySlo` evidence。只有實際全主機重啟事件、七主機 reachability、必要服務、public routes、資料 freshness、備份、edge fallback、VMware/Windows policy 與 188 hygiene 全部在同一個 10 分鐘 window 通過，才可宣稱 reboot recovery SLO 達標。
|
||||
|
||||
Reference in New Issue
Block a user