fix(reboot): bound windows99 vm discovery fallback
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 1m3s
CD Pipeline / build-and-deploy (push) Successful in 4m29s
CD Pipeline / post-deploy-checks (push) Successful in 1m48s

This commit is contained in:
Your Name
2026-07-03 17:04:05 +08:00
parent 10a9c7bb14
commit 40713b2f80
7 changed files with 45 additions and 1 deletions

View File

@@ -4,19 +4,22 @@
- production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已讀回新版 task run-state`active_blocker_count=9``runtime_metric_active_blocker_count=9``windows99_vmware_autostart_task_never_ran` 已進 active blockersscheduled task `last_task_result=267011` / `0x41303``last_run_time=1999-11-30T00:00:00`
- 110 no-secret artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-163131` 證明 collector 未讀 secret、未 remote write、未 VM power change但 host probe 同時顯示 `111 reachable=0`,而舊 Windows99 default target 卻是 `110 / 188 / 120 / 121 / 112`
- 修正 source-of-truth`windows99-vmware-autostart.ps1`、CLI scorecard、API scorecard fallback、console validator default、check-mode package tests 全部改為 `111 / 188 / 120 / 121 / 112`110 回到 P0 host / control-plane truth不再當 99 VMware guest default。
- 修正 runtime verifier`windows99-vmware-autostart.ps1` 預設不再遞迴掃 VM roots避免 111 VMX 缺失時讓 no-secret collector timeout`reboot-auto-recovery-slo-exporter.sh` 的 Windows99 fallback 只接受 alias set 完全符合 `111 / 188 / 120 / 121 / 112` 的 artifact避免舊 `110` artifact 回灌假讀回。
- live bounded collector 證明新版行為:`VMX alias=111 present=0``VM_POWER alias=111 vmx_present=0 running=0``VMWARE_AUTOSTART_CONFIG_READY=0``VMWARE_AUTOSTART_VERIFY_READY=0`,且 `secret_value_read=false``remote_write_performed=false``vm_power_change_performed=false`
- `FULL-STACK-COLD-START-SOP``REBOOT-RECOVERY-SOP``PUBLIC-MAINTENANCE-FALLBACK-RUNBOOK` 與 P0 workplan 已整合這次經驗,避免舊 `110` default 再遮蔽 `111 unreachable`
**已跑驗證**
- `python3.11 -m pytest scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_scorecard.py scripts/reboot-recovery/tests/test_windows99_console_verify_artifact_validator.py scripts/reboot-recovery/tests/test_windows99_vmware_verify_collector.py scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py -q -p no:cacheprovider``40 passed`
- `DATABASE_URL=postgresql+asyncpg://test:test@localhost/test PYTHONPATH=apps/api python3.11 -m pytest apps/api/tests/test_reboot_auto_recovery_slo_scorecard_api.py apps/api/tests/test_windows99_vmware_autostart_check_mode_package_api.py apps/api/tests/test_awoooi_priority_work_order_readback_api.py -q -p no:cacheprovider``44 passed`
- `python3.11 -m py_compile ...``bash -n scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh scripts/reboot-recovery/collect-windows99-vmware-verify.sh``python3.11 ops/runner/guard-gitea-runner-pressure.py --root .``git diff --check`:通過。
- `WINDOWS99_REMOTE_VERIFY_TIMEOUT=90 bash scripts/reboot-recovery/collect-windows99-vmware-verify.sh --collect --via-host wooo@192.168.0.110 --users wooo --timeout 8 --max-auth-users 1``rc=0`,新版 111 readback 成功。
**仍維持**
- 這不代表 10 分鐘重啟自動恢復 SLO 完成;目前 primary blocker 仍是 `reboot_event_required_host_unreachable`,且 `111` 仍需新版 Windows99 collector / production readback 確認。
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth沒有使用 GitHub / gh沒有啟動或關閉 VM沒有重啟 Windows host / VMware service沒有 Docker / Nginx / K3s / DB / firewall restart / reload沒有 DROP / TRUNCATE / restore / prune / delete / force push。
**下一步**
- commit / push 到 Gitea mainCD deploy 後受控安裝新版 110 SLO exporter 並重跑 no-secret collector讀回 production `required_vm_aliases` / `powered_off_aliases` / check-mode package 均以 111 為 Windows99 VM target
- commit / push 到 Gitea mainCD deploy 後受控安裝新版 110 SLO exporter 並重跑 no-secret collector讀回 production `missing_vmx_aliases=["111"]``powered_off_aliases` 含 111 且不含 110
## 2026-07-03 — 15:57 Windows99 VMware autostart task run-state 上卷

View File

@@ -82,6 +82,8 @@ v1.124 public maintenance route / edge blocker split ruleproduction API overl
v1.125 Windows99 autostart task run-state rule`windows99-vmware-autostart.ps1 -Mode Verify` 必須在既有 `VMWARE_AUTOSTART_TASK ... ok=...` 之外輸出 `VMWARE_AUTOSTART_TASK_DETAIL``VMWARE_AUTOSTART_TASK_ACTION`,至少包含 principal、run level、logon type、target script present、last run time、last task result、next run time、action execute / argument。`last_task_result=267011` / `0x41303``last_run_time=1999-11-30T00:00:00` 代表 task exists/enabled 但尚未跑過,若同時 `powered_off_aliases` 非空scorecard 必須新增 active blocker `windows99_vmware_autostart_task_never_ran`textfile exporter 必須輸出 `awoooi_windows99_vmware_autostart_task_*` metricsAPI overlay / action matrix 必須把 next action 指到 `run_windows99_vmware_autostart_check_mode_package_for_powered_off_aliases_then_rerun_no_secret_collector_no_vm_power_change`。2026-07-03 live no-secret via-110 readback 證明 `AWOOOI-Start-VMware-VMs` present/enabled/Ready`last_task_result=267011``last_run_time=1999-11-30T00:00:00`P0-4 真 target set 必須是 `111 / 188 / 120 / 121 / 112`,部署後重新收集 powered-off aliases不得再沿用舊 `110` default。此規則只授權 readback、metrics、alert routing 與 check-mode package不得直接啟動 VM、重啟 Windows host / VMware service、改 scheduled task、改 registry 或讀密碼 / secret。
v1.126 Windows99 bounded verifier / fallback compatibility rule`windows99-vmware-autostart.ps1 -Mode Verify` 預設不得遞迴掃描 VM roots111 VMX 缺失時要快速輸出 `VMX alias=111 present=0``VM_POWER alias=111 vmx_present=0 running=0`,深度尋找 VMX 必須走獨立 locator / check-mode lane。`reboot-auto-recovery-slo-exporter.sh` 的 Windows99 fallback artifact 必須與目前 default alias set `111 / 188 / 120 / 121 / 112` 完全相容;若舊 artifact 含 `110` 或缺 `111`,不得回灌 textfile / scorecard。這是避免 timeout 後用舊 110 artifact 假綠的防線;不授權 power change、relink、restore、host reboot 或 secret 讀取。
2026-07-02 110 control-path / Harbor recovery receipt rule若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/``401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh``recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335``Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` markernon110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blockernon110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。
2026-07-02 110 controlled CD lane fail-closed enforcer staging rule110 runner 壓力事故後legacy / generic runner 仍必須 fail-closed`awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host``awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner``CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail且 service `inactive``MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service若 registration 缺失readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail不要手工反覆補同一組 artifact。

View File

@@ -99,6 +99,8 @@ Production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已與 runtime me
2026-07-03 13:50 no-secret verifier overlay 曾以舊 default `110 / 188 / 120 / 121 / 112` 讀回 artifact2026-07-03 16:33 已判定該 default 與目前事故目標不一致。下一步必須部署新版 SLO exporter/collector 到 110、重跑 verifier service確認 validator default、Prometheus textfile、production scorecard 全部改讀 `111 / 188 / 120 / 121 / 112`。仍不得啟動 VM、relink、restore、重啟 host 或讀密碼。
2026-07-03 16:57 bounded verifier overlay新版 verifier 預設不再遞迴掃 VM roots避免 111 VMX 缺失時 timeout實測 via-110 no-secret collector 可回 `VMX alias=111 present=0``VM_POWER alias=111 vmx_present=0 running=0``VMWARE_AUTOSTART_CONFIG_READY=0`。SLO exporter fallback 只可使用 alias set 完全符合 `111 / 188 / 120 / 121 / 112` 的 artifact`110` artifact 必須被跳過,不得回灌成 runtime truth。
2026-07-03 14:10 runtime metric overlay110 最新 SLO textfile 已輸出 `active_blocker_count=7`,且 `awoooi_windows99_vmware_missing_vmx_alias` 沒有 alias gauge、`awoooi_windows99_vmware_powered_off_alias` 只列 `110 / 112 / 120 / 121 / 188`。production API 在 source 修正前仍可能因 stale committed snapshot 反加 Windows99 config / Windows Update policy 假 blocker新版 API 必須讀 `awoooi_windows99_vmware_autostart_config_ready` gauge並以 runtime metric overlay 清掉這兩個假 blocker。若 production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 仍顯示 `runtime_metric_active_blocker_count=7``active_blocker_count>7` 且多出的 blocker 是 Windows99 config / policy先部署 / 驗證 API overlay source不得回頭做 VMX locate、relink、power change 或 Windows policy apply。
### 五主機全貌

View File

@@ -20,6 +20,7 @@ POST_REBOOT_READINESS_TIMEOUT_SECONDS="${POST_REBOOT_READINESS_TIMEOUT_SECONDS:-
PUBLIC_MAINTENANCE_READBACK_TIMEOUT_SECONDS="${PUBLIC_MAINTENANCE_READBACK_TIMEOUT_SECONDS:-8}"
PUBLIC_MAINTENANCE_URLS="${PUBLIC_MAINTENANCE_URLS:-https://awoooi.wooo.work/api/v1/health https://awoooi.wooo.work/ https://stock.wooo.work/api/v1/system/freshness https://mo.wooo.work/health https://bitan.wooo.work/ https://www.tsenyang.com/}"
WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS="${WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS:-900}"
WINDOWS99_REQUIRED_VM_ALIASES="${WINDOWS99_REQUIRED_VM_ALIASES:-111 188 120 121 112}"
POST_REBOOT_READINESS_FALLBACK_MAX_AGE_SECONDS="${POST_REBOOT_READINESS_FALLBACK_MAX_AGE_SECONDS:-900}"
mkdir -p "$TEXTFILE_DIR" "$LOG_DIR"
@@ -323,11 +324,37 @@ if [ -f "$ROOT_DIR/scripts/reboot-recovery/windows99-management-channel-probe.py
fi
if [ -x "$ROOT_DIR/scripts/reboot-recovery/collect-windows99-vmware-verify.sh" ]; then
WINDOWS99_MAX_AUTH_USERS="${WINDOWS99_MAX_AUTH_USERS:-5}" \
WINDOWS99_REMOTE_VERIFY_TIMEOUT="${WINDOWS99_REMOTE_VERIFY_TIMEOUT:-90}" \
bash "$ROOT_DIR/scripts/reboot-recovery/collect-windows99-vmware-verify.sh" --collect \
>"$windows99_vmware_file" 2>"$artifact_dir/windows99-vmware-verify.err" \
|| true
fi
windows99_vmware_verify_alias_set_matches_current() {
local file="$1"
local alias
local observed
for alias in $WINDOWS99_REQUIRED_VM_ALIASES; do
grep -Eq "^(VMX|VM_POWER) alias=${alias}[[:space:]]" "$file" || return 1
done
while IFS= read -r observed; do
[ -n "$observed" ] || continue
case " $WINDOWS99_REQUIRED_VM_ALIASES " in
*" $observed "*) ;;
*) return 1 ;;
esac
done < <(awk '
/^(VMX|VM_POWER) alias=/ {
for (i = 1; i <= NF; i++) {
if ($i ~ /^alias=/) {
sub(/^alias=/, "", $i)
print $i
}
}
}
' "$file" | sort -u)
}
if ! [ -s "$windows99_vmware_file" ] || ! grep -q '^AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1$' "$windows99_vmware_file"; then
if [ -s "$windows99_vmware_file" ]; then
cp "$windows99_vmware_file" "$artifact_dir/windows99-vmware-verify.current-failed.txt" || true
@@ -340,6 +367,7 @@ if ! [ -s "$windows99_vmware_file" ] || ! grep -q '^AWOOOI_WINDOWS99_VMWARE_AUTO
[ "$candidate_path" = "$windows99_vmware_file" ] && continue
[ -s "$candidate_path" ] || continue
grep -q '^AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1$' "$candidate_path" || continue
windows99_vmware_verify_alias_set_matches_current "$candidate_path" || continue
candidate_epoch="${candidate_mtime%.*}"
candidate_age="$((now_epoch - candidate_epoch))"
[ "$candidate_age" -le "$WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS" ] || continue

View File

@@ -109,6 +109,9 @@ def test_exporter_projects_each_scorecard_blocker_to_textfile_metric() -> None:
assert "windows99-vmware-verify.txt" in text
assert "collect-windows99-vmware-verify.sh\" --collect" in text
assert "WINDOWS99_MAX_AUTH_USERS=\"${WINDOWS99_MAX_AUTH_USERS:-5}\"" in text
assert "WINDOWS99_REMOTE_VERIFY_TIMEOUT=\"${WINDOWS99_REMOTE_VERIFY_TIMEOUT:-90}\"" in text
assert "WINDOWS99_REQUIRED_VM_ALIASES=\"${WINDOWS99_REQUIRED_VM_ALIASES:-111 188 120 121 112}\"" in text
assert "windows99_vmware_verify_alias_set_matches_current" in text
assert "AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1" in text
assert "scorecard_args+=(--windows99-vmware-file" in text
assert "WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS" in text

View File

@@ -85,6 +85,7 @@ def test_reboot_p0_contract_covers_all_required_hosts_and_vmware_autostart() ->
assert "Host120Vmx" in windows99
assert "Host121Vmx" in windows99
assert "Host112Vmx" in windows99
assert "EnableRecursiveDiscovery" in windows99
assert "D:\\Documents\\Virtual Machines" in windows99
assert "D:\\Downloads" in windows99
assert '[string[]]$RequiredVmAliases = @("111", "188", "120", "121", "112")' in windows99

View File

@@ -9,6 +9,7 @@ param(
[string]$Host121Vmx = "",
[string]$Host112Vmx = "",
[string[]]$RequiredVmAliases = @("111", "188", "120", "121", "112"),
[switch]$EnableRecursiveDiscovery,
[string[]]$DiscoveryRoot = @(
"D:\Documents\Virtual Machines",
"D:\Downloads",
@@ -56,6 +57,10 @@ function Resolve-VmxPath {
}
}
if (-not $EnableRecursiveDiscovery) {
return ""
}
foreach ($root in $DiscoveryRoot) {
if (-not (Test-Path -LiteralPath $root)) {
continue