diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index b6fdcbef5..141a935fa 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -4,19 +4,22 @@ - production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已讀回新版 task run-state:`active_blocker_count=9`、`runtime_metric_active_blocker_count=9`、`windows99_vmware_autostart_task_never_ran` 已進 active blockers,scheduled task `last_task_result=267011` / `0x41303`、`last_run_time=1999-11-30T00:00:00`。 - 110 no-secret artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-163131` 證明 collector 未讀 secret、未 remote write、未 VM power change;但 host probe 同時顯示 `111 reachable=0`,而舊 Windows99 default target 卻是 `110 / 188 / 120 / 121 / 112`。 - 修正 source-of-truth:`windows99-vmware-autostart.ps1`、CLI scorecard、API scorecard fallback、console validator default、check-mode package tests 全部改為 `111 / 188 / 120 / 121 / 112`;110 回到 P0 host / control-plane truth,不再當 99 VMware guest default。 +- 修正 runtime verifier:`windows99-vmware-autostart.ps1` 預設不再遞迴掃 VM roots,避免 111 VMX 缺失時讓 no-secret collector timeout;`reboot-auto-recovery-slo-exporter.sh` 的 Windows99 fallback 只接受 alias set 完全符合 `111 / 188 / 120 / 121 / 112` 的 artifact,避免舊 `110` artifact 回灌假讀回。 +- live bounded collector 證明新版行為:`VMX alias=111 present=0`、`VM_POWER alias=111 vmx_present=0 running=0`、`VMWARE_AUTOSTART_CONFIG_READY=0`、`VMWARE_AUTOSTART_VERIFY_READY=0`,且 `secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`。 - `FULL-STACK-COLD-START-SOP`、`REBOOT-RECOVERY-SOP`、`PUBLIC-MAINTENANCE-FALLBACK-RUNBOOK` 與 P0 workplan 已整合這次經驗,避免舊 `110` default 再遮蔽 `111 unreachable`。 **已跑驗證**: - `python3.11 -m pytest scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_scorecard.py scripts/reboot-recovery/tests/test_windows99_console_verify_artifact_validator.py scripts/reboot-recovery/tests/test_windows99_vmware_verify_collector.py scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py -q -p no:cacheprovider`:`40 passed`。 - `DATABASE_URL=postgresql+asyncpg://test:test@localhost/test PYTHONPATH=apps/api python3.11 -m pytest apps/api/tests/test_reboot_auto_recovery_slo_scorecard_api.py apps/api/tests/test_windows99_vmware_autostart_check_mode_package_api.py apps/api/tests/test_awoooi_priority_work_order_readback_api.py -q -p no:cacheprovider`:`44 passed`。 - `python3.11 -m py_compile ...`、`bash -n scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh scripts/reboot-recovery/collect-windows99-vmware-verify.sh`、`python3.11 ops/runner/guard-gitea-runner-pressure.py --root .`、`git diff --check`:通過。 +- `WINDOWS99_REMOTE_VERIFY_TIMEOUT=90 bash scripts/reboot-recovery/collect-windows99-vmware-verify.sh --collect --via-host wooo@192.168.0.110 --users wooo --timeout 8 --max-auth-users 1`:`rc=0`,新版 111 readback 成功。 **仍維持**: - 這不代表 10 分鐘重啟自動恢復 SLO 完成;目前 primary blocker 仍是 `reboot_event_required_host_unreachable`,且 `111` 仍需新版 Windows99 collector / production readback 確認。 - 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth;沒有使用 GitHub / gh;沒有啟動或關閉 VM,沒有重啟 Windows host / VMware service,沒有 Docker / Nginx / K3s / DB / firewall restart / reload,沒有 DROP / TRUNCATE / restore / prune / delete / force push。 **下一步**: -- commit / push 到 Gitea main;CD deploy 後受控安裝新版 110 SLO exporter 並重跑 no-secret collector,讀回 production `required_vm_aliases` / `powered_off_aliases` / check-mode package 均以 111 為 Windows99 VM target。 +- commit / push 到 Gitea main;CD deploy 後受控安裝新版 110 SLO exporter 並重跑 no-secret collector,讀回 production `missing_vmx_aliases=["111"]`、`powered_off_aliases` 含 111 且不含 110。 ## 2026-07-03 — 15:57 Windows99 VMware autostart task run-state 上卷 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 3dbe0585c..2b856304b 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -82,6 +82,8 @@ v1.124 public maintenance route / edge blocker split rule:production API overl v1.125 Windows99 autostart task run-state rule:`windows99-vmware-autostart.ps1 -Mode Verify` 必須在既有 `VMWARE_AUTOSTART_TASK ... ok=...` 之外輸出 `VMWARE_AUTOSTART_TASK_DETAIL` 與 `VMWARE_AUTOSTART_TASK_ACTION`,至少包含 principal、run level、logon type、target script present、last run time、last task result、next run time、action execute / argument。`last_task_result=267011` / `0x41303` 或 `last_run_time=1999-11-30T00:00:00` 代表 task exists/enabled 但尚未跑過,若同時 `powered_off_aliases` 非空,scorecard 必須新增 active blocker `windows99_vmware_autostart_task_never_ran`,textfile exporter 必須輸出 `awoooi_windows99_vmware_autostart_task_*` metrics,API overlay / action matrix 必須把 next action 指到 `run_windows99_vmware_autostart_check_mode_package_for_powered_off_aliases_then_rerun_no_secret_collector_no_vm_power_change`。2026-07-03 live no-secret via-110 readback 證明 `AWOOOI-Start-VMware-VMs` present/enabled/Ready,但 `last_task_result=267011`、`last_run_time=1999-11-30T00:00:00`;P0-4 真 target set 必須是 `111 / 188 / 120 / 121 / 112`,部署後重新收集 powered-off aliases,不得再沿用舊 `110` default。此規則只授權 readback、metrics、alert routing 與 check-mode package;不得直接啟動 VM、重啟 Windows host / VMware service、改 scheduled task、改 registry 或讀密碼 / secret。 +v1.126 Windows99 bounded verifier / fallback compatibility rule:`windows99-vmware-autostart.ps1 -Mode Verify` 預設不得遞迴掃描 VM roots;111 VMX 缺失時要快速輸出 `VMX alias=111 present=0` 與 `VM_POWER alias=111 vmx_present=0 running=0`,深度尋找 VMX 必須走獨立 locator / check-mode lane。`reboot-auto-recovery-slo-exporter.sh` 的 Windows99 fallback artifact 必須與目前 default alias set `111 / 188 / 120 / 121 / 112` 完全相容;若舊 artifact 含 `110` 或缺 `111`,不得回灌 textfile / scorecard。這是避免 timeout 後用舊 110 artifact 假綠的防線;不授權 power change、relink、restore、host reboot 或 secret 讀取。 + 2026-07-02 110 control-path / Harbor recovery receipt rule:若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure,但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata,不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier,並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示:public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`;active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure,以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker,non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker;non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。 2026-07-02 110 controlled CD lane fail-closed enforcer staging rule:110 runner 壓力事故後,legacy / generic runner 仍必須 fail-closed;但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail,且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit,並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service;若 registration 缺失,readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail,不要手工反覆補同一組 artifact。 diff --git a/docs/runbooks/REBOOT-RECOVERY-SOP.md b/docs/runbooks/REBOOT-RECOVERY-SOP.md index be9d2bce8..d87e6591f 100644 --- a/docs/runbooks/REBOOT-RECOVERY-SOP.md +++ b/docs/runbooks/REBOOT-RECOVERY-SOP.md @@ -99,6 +99,8 @@ Production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已與 runtime me 2026-07-03 13:50 no-secret verifier overlay 曾以舊 default `110 / 188 / 120 / 121 / 112` 讀回 artifact;2026-07-03 16:33 已判定該 default 與目前事故目標不一致。下一步必須部署新版 SLO exporter/collector 到 110、重跑 verifier service,確認 validator default、Prometheus textfile、production scorecard 全部改讀 `111 / 188 / 120 / 121 / 112`。仍不得啟動 VM、relink、restore、重啟 host 或讀密碼。 +2026-07-03 16:57 bounded verifier overlay:新版 verifier 預設不再遞迴掃 VM roots,避免 111 VMX 缺失時 timeout;實測 via-110 no-secret collector 可回 `VMX alias=111 present=0`、`VM_POWER alias=111 vmx_present=0 running=0`、`VMWARE_AUTOSTART_CONFIG_READY=0`。SLO exporter fallback 只可使用 alias set 完全符合 `111 / 188 / 120 / 121 / 112` 的 artifact;舊 `110` artifact 必須被跳過,不得回灌成 runtime truth。 + 2026-07-03 14:10 runtime metric overlay:110 最新 SLO textfile 已輸出 `active_blocker_count=7`,且 `awoooi_windows99_vmware_missing_vmx_alias` 沒有 alias gauge、`awoooi_windows99_vmware_powered_off_alias` 只列 `110 / 112 / 120 / 121 / 188`。production API 在 source 修正前仍可能因 stale committed snapshot 反加 Windows99 config / Windows Update policy 假 blocker;新版 API 必須讀 `awoooi_windows99_vmware_autostart_config_ready` gauge,並以 runtime metric overlay 清掉這兩個假 blocker。若 production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 仍顯示 `runtime_metric_active_blocker_count=7` 但 `active_blocker_count>7` 且多出的 blocker 是 Windows99 config / policy,先部署 / 驗證 API overlay source,不得回頭做 VMX locate、relink、power change 或 Windows policy apply。 ### 五主機全貌 diff --git a/scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh b/scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh index 6d908bde6..c0b4a5b35 100755 --- a/scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh +++ b/scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh @@ -20,6 +20,7 @@ POST_REBOOT_READINESS_TIMEOUT_SECONDS="${POST_REBOOT_READINESS_TIMEOUT_SECONDS:- PUBLIC_MAINTENANCE_READBACK_TIMEOUT_SECONDS="${PUBLIC_MAINTENANCE_READBACK_TIMEOUT_SECONDS:-8}" PUBLIC_MAINTENANCE_URLS="${PUBLIC_MAINTENANCE_URLS:-https://awoooi.wooo.work/api/v1/health https://awoooi.wooo.work/ https://stock.wooo.work/api/v1/system/freshness https://mo.wooo.work/health https://bitan.wooo.work/ https://www.tsenyang.com/}" WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS="${WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS:-900}" +WINDOWS99_REQUIRED_VM_ALIASES="${WINDOWS99_REQUIRED_VM_ALIASES:-111 188 120 121 112}" POST_REBOOT_READINESS_FALLBACK_MAX_AGE_SECONDS="${POST_REBOOT_READINESS_FALLBACK_MAX_AGE_SECONDS:-900}" mkdir -p "$TEXTFILE_DIR" "$LOG_DIR" @@ -323,11 +324,37 @@ if [ -f "$ROOT_DIR/scripts/reboot-recovery/windows99-management-channel-probe.py fi if [ -x "$ROOT_DIR/scripts/reboot-recovery/collect-windows99-vmware-verify.sh" ]; then WINDOWS99_MAX_AUTH_USERS="${WINDOWS99_MAX_AUTH_USERS:-5}" \ + WINDOWS99_REMOTE_VERIFY_TIMEOUT="${WINDOWS99_REMOTE_VERIFY_TIMEOUT:-90}" \ bash "$ROOT_DIR/scripts/reboot-recovery/collect-windows99-vmware-verify.sh" --collect \ >"$windows99_vmware_file" 2>"$artifact_dir/windows99-vmware-verify.err" \ || true fi +windows99_vmware_verify_alias_set_matches_current() { + local file="$1" + local alias + local observed + for alias in $WINDOWS99_REQUIRED_VM_ALIASES; do + grep -Eq "^(VMX|VM_POWER) alias=${alias}[[:space:]]" "$file" || return 1 + done + while IFS= read -r observed; do + [ -n "$observed" ] || continue + case " $WINDOWS99_REQUIRED_VM_ALIASES " in + *" $observed "*) ;; + *) return 1 ;; + esac + done < <(awk ' + /^(VMX|VM_POWER) alias=/ { + for (i = 1; i <= NF; i++) { + if ($i ~ /^alias=/) { + sub(/^alias=/, "", $i) + print $i + } + } + } + ' "$file" | sort -u) +} + if ! [ -s "$windows99_vmware_file" ] || ! grep -q '^AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1$' "$windows99_vmware_file"; then if [ -s "$windows99_vmware_file" ]; then cp "$windows99_vmware_file" "$artifact_dir/windows99-vmware-verify.current-failed.txt" || true @@ -340,6 +367,7 @@ if ! [ -s "$windows99_vmware_file" ] || ! grep -q '^AWOOOI_WINDOWS99_VMWARE_AUTO [ "$candidate_path" = "$windows99_vmware_file" ] && continue [ -s "$candidate_path" ] || continue grep -q '^AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1$' "$candidate_path" || continue + windows99_vmware_verify_alias_set_matches_current "$candidate_path" || continue candidate_epoch="${candidate_mtime%.*}" candidate_age="$((now_epoch - candidate_epoch))" [ "$candidate_age" -le "$WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS" ] || continue diff --git a/scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_installer.py b/scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_installer.py index e97c6b10a..dc9101775 100644 --- a/scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_installer.py +++ b/scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_installer.py @@ -109,6 +109,9 @@ def test_exporter_projects_each_scorecard_blocker_to_textfile_metric() -> None: assert "windows99-vmware-verify.txt" in text assert "collect-windows99-vmware-verify.sh\" --collect" in text assert "WINDOWS99_MAX_AUTH_USERS=\"${WINDOWS99_MAX_AUTH_USERS:-5}\"" in text + assert "WINDOWS99_REMOTE_VERIFY_TIMEOUT=\"${WINDOWS99_REMOTE_VERIFY_TIMEOUT:-90}\"" in text + assert "WINDOWS99_REQUIRED_VM_ALIASES=\"${WINDOWS99_REQUIRED_VM_ALIASES:-111 188 120 121 112}\"" in text + assert "windows99_vmware_verify_alias_set_matches_current" in text assert "AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1" in text assert "scorecard_args+=(--windows99-vmware-file" in text assert "WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS" in text diff --git a/scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py b/scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py index b713cc067..6411163be 100644 --- a/scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py +++ b/scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py @@ -85,6 +85,7 @@ def test_reboot_p0_contract_covers_all_required_hosts_and_vmware_autostart() -> assert "Host120Vmx" in windows99 assert "Host121Vmx" in windows99 assert "Host112Vmx" in windows99 + assert "EnableRecursiveDiscovery" in windows99 assert "D:\\Documents\\Virtual Machines" in windows99 assert "D:\\Downloads" in windows99 assert '[string[]]$RequiredVmAliases = @("111", "188", "120", "121", "112")' in windows99 diff --git a/scripts/reboot-recovery/windows99-vmware-autostart.ps1 b/scripts/reboot-recovery/windows99-vmware-autostart.ps1 index 906590bdc..6c5e22257 100644 --- a/scripts/reboot-recovery/windows99-vmware-autostart.ps1 +++ b/scripts/reboot-recovery/windows99-vmware-autostart.ps1 @@ -9,6 +9,7 @@ param( [string]$Host121Vmx = "", [string]$Host112Vmx = "", [string[]]$RequiredVmAliases = @("111", "188", "120", "121", "112"), + [switch]$EnableRecursiveDiscovery, [string[]]$DiscoveryRoot = @( "D:\Documents\Virtual Machines", "D:\Downloads", @@ -56,6 +57,10 @@ function Resolve-VmxPath { } } + if (-not $EnableRecursiveDiscovery) { + return "" + } + foreach ($root in $DiscoveryRoot) { if (-not (Test-Path -LiteralPath $root)) { continue