fix(reboot): bound windows99 vm discovery fallback
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 1m3s
CD Pipeline / build-and-deploy (push) Successful in 4m29s
CD Pipeline / post-deploy-checks (push) Successful in 1m48s
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 1m3s
CD Pipeline / build-and-deploy (push) Successful in 4m29s
CD Pipeline / post-deploy-checks (push) Successful in 1m48s
This commit is contained in:
@@ -4,19 +4,22 @@
|
||||
- production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已讀回新版 task run-state:`active_blocker_count=9`、`runtime_metric_active_blocker_count=9`、`windows99_vmware_autostart_task_never_ran` 已進 active blockers,scheduled task `last_task_result=267011` / `0x41303`、`last_run_time=1999-11-30T00:00:00`。
|
||||
- 110 no-secret artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-163131` 證明 collector 未讀 secret、未 remote write、未 VM power change;但 host probe 同時顯示 `111 reachable=0`,而舊 Windows99 default target 卻是 `110 / 188 / 120 / 121 / 112`。
|
||||
- 修正 source-of-truth:`windows99-vmware-autostart.ps1`、CLI scorecard、API scorecard fallback、console validator default、check-mode package tests 全部改為 `111 / 188 / 120 / 121 / 112`;110 回到 P0 host / control-plane truth,不再當 99 VMware guest default。
|
||||
- 修正 runtime verifier:`windows99-vmware-autostart.ps1` 預設不再遞迴掃 VM roots,避免 111 VMX 缺失時讓 no-secret collector timeout;`reboot-auto-recovery-slo-exporter.sh` 的 Windows99 fallback 只接受 alias set 完全符合 `111 / 188 / 120 / 121 / 112` 的 artifact,避免舊 `110` artifact 回灌假讀回。
|
||||
- live bounded collector 證明新版行為:`VMX alias=111 present=0`、`VM_POWER alias=111 vmx_present=0 running=0`、`VMWARE_AUTOSTART_CONFIG_READY=0`、`VMWARE_AUTOSTART_VERIFY_READY=0`,且 `secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`。
|
||||
- `FULL-STACK-COLD-START-SOP`、`REBOOT-RECOVERY-SOP`、`PUBLIC-MAINTENANCE-FALLBACK-RUNBOOK` 與 P0 workplan 已整合這次經驗,避免舊 `110` default 再遮蔽 `111 unreachable`。
|
||||
|
||||
**已跑驗證**:
|
||||
- `python3.11 -m pytest scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_scorecard.py scripts/reboot-recovery/tests/test_windows99_console_verify_artifact_validator.py scripts/reboot-recovery/tests/test_windows99_vmware_verify_collector.py scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py -q -p no:cacheprovider`:`40 passed`。
|
||||
- `DATABASE_URL=postgresql+asyncpg://test:test@localhost/test PYTHONPATH=apps/api python3.11 -m pytest apps/api/tests/test_reboot_auto_recovery_slo_scorecard_api.py apps/api/tests/test_windows99_vmware_autostart_check_mode_package_api.py apps/api/tests/test_awoooi_priority_work_order_readback_api.py -q -p no:cacheprovider`:`44 passed`。
|
||||
- `python3.11 -m py_compile ...`、`bash -n scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh scripts/reboot-recovery/collect-windows99-vmware-verify.sh`、`python3.11 ops/runner/guard-gitea-runner-pressure.py --root .`、`git diff --check`:通過。
|
||||
- `WINDOWS99_REMOTE_VERIFY_TIMEOUT=90 bash scripts/reboot-recovery/collect-windows99-vmware-verify.sh --collect --via-host wooo@192.168.0.110 --users wooo --timeout 8 --max-auth-users 1`:`rc=0`,新版 111 readback 成功。
|
||||
|
||||
**仍維持**:
|
||||
- 這不代表 10 分鐘重啟自動恢復 SLO 完成;目前 primary blocker 仍是 `reboot_event_required_host_unreachable`,且 `111` 仍需新版 Windows99 collector / production readback 確認。
|
||||
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth;沒有使用 GitHub / gh;沒有啟動或關閉 VM,沒有重啟 Windows host / VMware service,沒有 Docker / Nginx / K3s / DB / firewall restart / reload,沒有 DROP / TRUNCATE / restore / prune / delete / force push。
|
||||
|
||||
**下一步**:
|
||||
- commit / push 到 Gitea main;CD deploy 後受控安裝新版 110 SLO exporter 並重跑 no-secret collector,讀回 production `required_vm_aliases` / `powered_off_aliases` / check-mode package 均以 111 為 Windows99 VM target。
|
||||
- commit / push 到 Gitea main;CD deploy 後受控安裝新版 110 SLO exporter 並重跑 no-secret collector,讀回 production `missing_vmx_aliases=["111"]`、`powered_off_aliases` 含 111 且不含 110。
|
||||
|
||||
## 2026-07-03 — 15:57 Windows99 VMware autostart task run-state 上卷
|
||||
|
||||
|
||||
@@ -82,6 +82,8 @@ v1.124 public maintenance route / edge blocker split rule:production API overl
|
||||
|
||||
v1.125 Windows99 autostart task run-state rule:`windows99-vmware-autostart.ps1 -Mode Verify` 必須在既有 `VMWARE_AUTOSTART_TASK ... ok=...` 之外輸出 `VMWARE_AUTOSTART_TASK_DETAIL` 與 `VMWARE_AUTOSTART_TASK_ACTION`,至少包含 principal、run level、logon type、target script present、last run time、last task result、next run time、action execute / argument。`last_task_result=267011` / `0x41303` 或 `last_run_time=1999-11-30T00:00:00` 代表 task exists/enabled 但尚未跑過,若同時 `powered_off_aliases` 非空,scorecard 必須新增 active blocker `windows99_vmware_autostart_task_never_ran`,textfile exporter 必須輸出 `awoooi_windows99_vmware_autostart_task_*` metrics,API overlay / action matrix 必須把 next action 指到 `run_windows99_vmware_autostart_check_mode_package_for_powered_off_aliases_then_rerun_no_secret_collector_no_vm_power_change`。2026-07-03 live no-secret via-110 readback 證明 `AWOOOI-Start-VMware-VMs` present/enabled/Ready,但 `last_task_result=267011`、`last_run_time=1999-11-30T00:00:00`;P0-4 真 target set 必須是 `111 / 188 / 120 / 121 / 112`,部署後重新收集 powered-off aliases,不得再沿用舊 `110` default。此規則只授權 readback、metrics、alert routing 與 check-mode package;不得直接啟動 VM、重啟 Windows host / VMware service、改 scheduled task、改 registry 或讀密碼 / secret。
|
||||
|
||||
v1.126 Windows99 bounded verifier / fallback compatibility rule:`windows99-vmware-autostart.ps1 -Mode Verify` 預設不得遞迴掃描 VM roots;111 VMX 缺失時要快速輸出 `VMX alias=111 present=0` 與 `VM_POWER alias=111 vmx_present=0 running=0`,深度尋找 VMX 必須走獨立 locator / check-mode lane。`reboot-auto-recovery-slo-exporter.sh` 的 Windows99 fallback artifact 必須與目前 default alias set `111 / 188 / 120 / 121 / 112` 完全相容;若舊 artifact 含 `110` 或缺 `111`,不得回灌 textfile / scorecard。這是避免 timeout 後用舊 110 artifact 假綠的防線;不授權 power change、relink、restore、host reboot 或 secret 讀取。
|
||||
|
||||
2026-07-02 110 control-path / Harbor recovery receipt rule:若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure,但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata,不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier,並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示:public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`;active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure,以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker,non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker;non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。
|
||||
|
||||
2026-07-02 110 controlled CD lane fail-closed enforcer staging rule:110 runner 壓力事故後,legacy / generic runner 仍必須 fail-closed;但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail,且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit,並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service;若 registration 缺失,readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail,不要手工反覆補同一組 artifact。
|
||||
|
||||
@@ -99,6 +99,8 @@ Production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已與 runtime me
|
||||
|
||||
2026-07-03 13:50 no-secret verifier overlay 曾以舊 default `110 / 188 / 120 / 121 / 112` 讀回 artifact;2026-07-03 16:33 已判定該 default 與目前事故目標不一致。下一步必須部署新版 SLO exporter/collector 到 110、重跑 verifier service,確認 validator default、Prometheus textfile、production scorecard 全部改讀 `111 / 188 / 120 / 121 / 112`。仍不得啟動 VM、relink、restore、重啟 host 或讀密碼。
|
||||
|
||||
2026-07-03 16:57 bounded verifier overlay:新版 verifier 預設不再遞迴掃 VM roots,避免 111 VMX 缺失時 timeout;實測 via-110 no-secret collector 可回 `VMX alias=111 present=0`、`VM_POWER alias=111 vmx_present=0 running=0`、`VMWARE_AUTOSTART_CONFIG_READY=0`。SLO exporter fallback 只可使用 alias set 完全符合 `111 / 188 / 120 / 121 / 112` 的 artifact;舊 `110` artifact 必須被跳過,不得回灌成 runtime truth。
|
||||
|
||||
2026-07-03 14:10 runtime metric overlay:110 最新 SLO textfile 已輸出 `active_blocker_count=7`,且 `awoooi_windows99_vmware_missing_vmx_alias` 沒有 alias gauge、`awoooi_windows99_vmware_powered_off_alias` 只列 `110 / 112 / 120 / 121 / 188`。production API 在 source 修正前仍可能因 stale committed snapshot 反加 Windows99 config / Windows Update policy 假 blocker;新版 API 必須讀 `awoooi_windows99_vmware_autostart_config_ready` gauge,並以 runtime metric overlay 清掉這兩個假 blocker。若 production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 仍顯示 `runtime_metric_active_blocker_count=7` 但 `active_blocker_count>7` 且多出的 blocker 是 Windows99 config / policy,先部署 / 驗證 API overlay source,不得回頭做 VMX locate、relink、power change 或 Windows policy apply。
|
||||
|
||||
### 五主機全貌
|
||||
|
||||
@@ -20,6 +20,7 @@ POST_REBOOT_READINESS_TIMEOUT_SECONDS="${POST_REBOOT_READINESS_TIMEOUT_SECONDS:-
|
||||
PUBLIC_MAINTENANCE_READBACK_TIMEOUT_SECONDS="${PUBLIC_MAINTENANCE_READBACK_TIMEOUT_SECONDS:-8}"
|
||||
PUBLIC_MAINTENANCE_URLS="${PUBLIC_MAINTENANCE_URLS:-https://awoooi.wooo.work/api/v1/health https://awoooi.wooo.work/ https://stock.wooo.work/api/v1/system/freshness https://mo.wooo.work/health https://bitan.wooo.work/ https://www.tsenyang.com/}"
|
||||
WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS="${WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS:-900}"
|
||||
WINDOWS99_REQUIRED_VM_ALIASES="${WINDOWS99_REQUIRED_VM_ALIASES:-111 188 120 121 112}"
|
||||
POST_REBOOT_READINESS_FALLBACK_MAX_AGE_SECONDS="${POST_REBOOT_READINESS_FALLBACK_MAX_AGE_SECONDS:-900}"
|
||||
|
||||
mkdir -p "$TEXTFILE_DIR" "$LOG_DIR"
|
||||
@@ -323,11 +324,37 @@ if [ -f "$ROOT_DIR/scripts/reboot-recovery/windows99-management-channel-probe.py
|
||||
fi
|
||||
if [ -x "$ROOT_DIR/scripts/reboot-recovery/collect-windows99-vmware-verify.sh" ]; then
|
||||
WINDOWS99_MAX_AUTH_USERS="${WINDOWS99_MAX_AUTH_USERS:-5}" \
|
||||
WINDOWS99_REMOTE_VERIFY_TIMEOUT="${WINDOWS99_REMOTE_VERIFY_TIMEOUT:-90}" \
|
||||
bash "$ROOT_DIR/scripts/reboot-recovery/collect-windows99-vmware-verify.sh" --collect \
|
||||
>"$windows99_vmware_file" 2>"$artifact_dir/windows99-vmware-verify.err" \
|
||||
|| true
|
||||
fi
|
||||
|
||||
windows99_vmware_verify_alias_set_matches_current() {
|
||||
local file="$1"
|
||||
local alias
|
||||
local observed
|
||||
for alias in $WINDOWS99_REQUIRED_VM_ALIASES; do
|
||||
grep -Eq "^(VMX|VM_POWER) alias=${alias}[[:space:]]" "$file" || return 1
|
||||
done
|
||||
while IFS= read -r observed; do
|
||||
[ -n "$observed" ] || continue
|
||||
case " $WINDOWS99_REQUIRED_VM_ALIASES " in
|
||||
*" $observed "*) ;;
|
||||
*) return 1 ;;
|
||||
esac
|
||||
done < <(awk '
|
||||
/^(VMX|VM_POWER) alias=/ {
|
||||
for (i = 1; i <= NF; i++) {
|
||||
if ($i ~ /^alias=/) {
|
||||
sub(/^alias=/, "", $i)
|
||||
print $i
|
||||
}
|
||||
}
|
||||
}
|
||||
' "$file" | sort -u)
|
||||
}
|
||||
|
||||
if ! [ -s "$windows99_vmware_file" ] || ! grep -q '^AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1$' "$windows99_vmware_file"; then
|
||||
if [ -s "$windows99_vmware_file" ]; then
|
||||
cp "$windows99_vmware_file" "$artifact_dir/windows99-vmware-verify.current-failed.txt" || true
|
||||
@@ -340,6 +367,7 @@ if ! [ -s "$windows99_vmware_file" ] || ! grep -q '^AWOOOI_WINDOWS99_VMWARE_AUTO
|
||||
[ "$candidate_path" = "$windows99_vmware_file" ] && continue
|
||||
[ -s "$candidate_path" ] || continue
|
||||
grep -q '^AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1$' "$candidate_path" || continue
|
||||
windows99_vmware_verify_alias_set_matches_current "$candidate_path" || continue
|
||||
candidate_epoch="${candidate_mtime%.*}"
|
||||
candidate_age="$((now_epoch - candidate_epoch))"
|
||||
[ "$candidate_age" -le "$WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS" ] || continue
|
||||
|
||||
@@ -109,6 +109,9 @@ def test_exporter_projects_each_scorecard_blocker_to_textfile_metric() -> None:
|
||||
assert "windows99-vmware-verify.txt" in text
|
||||
assert "collect-windows99-vmware-verify.sh\" --collect" in text
|
||||
assert "WINDOWS99_MAX_AUTH_USERS=\"${WINDOWS99_MAX_AUTH_USERS:-5}\"" in text
|
||||
assert "WINDOWS99_REMOTE_VERIFY_TIMEOUT=\"${WINDOWS99_REMOTE_VERIFY_TIMEOUT:-90}\"" in text
|
||||
assert "WINDOWS99_REQUIRED_VM_ALIASES=\"${WINDOWS99_REQUIRED_VM_ALIASES:-111 188 120 121 112}\"" in text
|
||||
assert "windows99_vmware_verify_alias_set_matches_current" in text
|
||||
assert "AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1" in text
|
||||
assert "scorecard_args+=(--windows99-vmware-file" in text
|
||||
assert "WINDOWS99_VMWARE_FALLBACK_MAX_AGE_SECONDS" in text
|
||||
|
||||
@@ -85,6 +85,7 @@ def test_reboot_p0_contract_covers_all_required_hosts_and_vmware_autostart() ->
|
||||
assert "Host120Vmx" in windows99
|
||||
assert "Host121Vmx" in windows99
|
||||
assert "Host112Vmx" in windows99
|
||||
assert "EnableRecursiveDiscovery" in windows99
|
||||
assert "D:\\Documents\\Virtual Machines" in windows99
|
||||
assert "D:\\Downloads" in windows99
|
||||
assert '[string[]]$RequiredVmAliases = @("111", "188", "120", "121", "112")' in windows99
|
||||
|
||||
@@ -9,6 +9,7 @@ param(
|
||||
[string]$Host121Vmx = "",
|
||||
[string]$Host112Vmx = "",
|
||||
[string[]]$RequiredVmAliases = @("111", "188", "120", "121", "112"),
|
||||
[switch]$EnableRecursiveDiscovery,
|
||||
[string[]]$DiscoveryRoot = @(
|
||||
"D:\Documents\Virtual Machines",
|
||||
"D:\Downloads",
|
||||
@@ -56,6 +57,10 @@ function Resolve-VmxPath {
|
||||
}
|
||||
}
|
||||
|
||||
if (-not $EnableRecursiveDiscovery) {
|
||||
return ""
|
||||
}
|
||||
|
||||
foreach ($root in $DiscoveryRoot) {
|
||||
if (-not (Test-Path -LiteralPath $root)) {
|
||||
continue
|
||||
|
||||
Reference in New Issue
Block a user