fix(reboot): support windows99 verifier via host
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 1m1s
CD Pipeline / build-and-deploy (push) Successful in 4m28s
CD Pipeline / post-deploy-checks (push) Successful in 1m45s
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 1m1s
CD Pipeline / build-and-deploy (push) Successful in 4m28s
CD Pipeline / post-deploy-checks (push) Successful in 1m45s
This commit is contained in:
@@ -7,15 +7,18 @@
|
||||
- `scripts/reboot-recovery/windows99-vmware-autostart.ps1` default target 改回 `110 / 188 / 120 / 121 / 112`;若未來提供 explicit `Host111Vmx`,會自動將 `111` 納入 VM order。
|
||||
- `reboot-auto-recovery-slo-scorecard` API / CLI fallback 同步改為 `110 / 188 / 120 / 121 / 112`;CLI parser 若 verifier stdout 已列出 `VMX alias=...` / `VM_POWER alias=...`,改以實際 verifier aliases 為準,保留 explicit `111` artifact / future override 的 repair lane。
|
||||
- `windows99_vmx_source_repair_package.vm_rows` 改為 default required aliases 與 runtime missing / powered-off aliases 的 union,避免 Prometheus runtime 明確送進來的 alias 被 package row 漏掉。
|
||||
- `FULL-STACK-COLD-START-SOP` 升到 v1.118,`REBOOT-RECOVERY-SOP` 升到 v5.10,固定記錄 Windows99 default VM target 與 111 required host 的分層。
|
||||
- `collect-windows99-vmware-verify.sh` 補上 `--via-host`,讓本工作站可走 `wooo@192.168.0.110` 的 no-secret path 收 99 Verify stdout;`validate-windows99-console-verify-artifact.py` default required aliases 同步改為 `110 / 188 / 120 / 121 / 112`,explicit 111 artifact 必須用 `--required-aliases` 指定。
|
||||
- Live via-host artifact `/tmp/awoooi-windows99-vmware-verify-via110-20260703134830.txt` 已驗證:collector exit `0`,validator `valid_console_verify_artifact_vmware_not_ready`,`missing_vmx_aliases=[]`,`powered_off_aliases=110,188,120,121,112`,`config_ready=True`,`windows_update_no_auto_reboot_ready=True`,`verify_ready=False`。因此目前 Windows99 lane 應從 `111 VMX source repair` 收斂為 `VM power/readback not ready`。
|
||||
- `FULL-STACK-COLD-START-SOP` 升到 v1.119,`REBOOT-RECOVERY-SOP` 升到 v5.11,固定記錄 Windows99 default VM target、111 required host 分層、via-host no-secret verifier 與 validator default。
|
||||
|
||||
**已跑驗證**:
|
||||
- `python3.11 -m py_compile apps/api/src/services/reboot_auto_recovery_slo_scorecard.py scripts/reboot-recovery/reboot-auto-recovery-slo-scorecard.py`:通過。
|
||||
- `DATABASE_URL=postgresql+asyncpg://test:test@localhost/test PYTHONPATH=apps/api python3.11 -m pytest scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_scorecard.py scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py scripts/reboot-recovery/tests/test_windows99_vmx_source_repair_package.py apps/api/tests/test_reboot_auto_recovery_slo_scorecard_api.py apps/api/tests/test_windows99_vmx_source_repair_package_api.py apps/api/tests/test_windows99_vmx_source_locator_readback_api.py apps/api/tests/test_awoooi_priority_work_order_readback_api.py -q -p no:cacheprovider`:`71 passed`。
|
||||
- `DATABASE_URL=postgresql+asyncpg://test:test@localhost/test PYTHONPATH=apps/api python3.11 -m pytest scripts/reboot-recovery/tests/test_windows99_console_verify_artifact_validator.py scripts/reboot-recovery/tests/test_windows99_vmware_verify_collector.py scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_scorecard.py scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py apps/api/tests/test_reboot_auto_recovery_slo_scorecard_api.py -q -p no:cacheprovider`:`57 passed`。
|
||||
|
||||
**仍維持**:
|
||||
- 此輪只修 source/readback truth,不執行 Windows task apply、不 relink、不 restore、不啟動 VM、不重啟 host / service、不碰 Docker / Nginx / K3s / DB / firewall;未讀 secret / token / `.env` / raw sessions / SQLite / auth,未使用 GitHub / gh,未 workflow_dispatch。
|
||||
- P0-006 尚未完成:要重新部署後讀回 production,並重跑 no-secret verifier / scorecard,確認 Windows99 VM target truth 不再把 111 VMX 當預設 blocker;111 仍需作為 required host 可達性與服務 freshness 處理。
|
||||
- P0-006 尚未完成:要部署新版 collector / validator / scorecard 到 110,重跑 verifier service,讓 Prometheus / production scorecard 清掉舊 `windows99_vmware_vmx_missing` runtime metric;111 仍需作為 required host 可達性與服務 freshness 處理。
|
||||
|
||||
## 2026-07-03 — 13:12 Windows99 VMX relink dry-run / identity gate 上卷
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# AWOOOI 全棧冷啟動與主機重啟 SOP
|
||||
|
||||
> Version: v1.118
|
||||
> Version: v1.119
|
||||
> Last updated: 2026-07-03 Asia/Taipei
|
||||
> Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號,但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal;此納入不代表授權任何破壞性 runtime apply。
|
||||
|
||||
@@ -68,6 +68,8 @@ v1.117 Windows99 VMX candidate confirmation gate rule:locator receipt 顯示 `
|
||||
|
||||
v1.118 Windows99 VM target source-of-truth reconciliation rule:99 VMware autostart 的預設 target 不得與 2026-07-01 runtime apply truth 分叉。repo source、API scorecard fallback、CLI scorecard fallback 必須一致使用 `110 / 188 / 120 / 121 / 112`;`111` 仍是全主機 reboot SLO 的 required host,但不是 99 VMware 預設 VMX source。若 verifier stdout 已列出 `VMX alias=...` / `VM_POWER alias=...`,scorecard 必須以 verifier 實際 alias 集合為準,讓舊的 explicit `111` artifact 或未來 `Host111Vmx` override 仍可走 `111` repair lane。`windows99_vmx_source_repair_package.vm_rows` 必須是 default required aliases 與 runtime missing / powered-off aliases 的 union,避免 runtime 明確送進來的 alias 被 UI/API row 漏掉。此規則只修 source-of-truth / readback;不授權改 Windows task、relink、restore、啟動 VM 或重啟 host。
|
||||
|
||||
v1.119 Windows99 no-secret verifier via-host / validator rule:本工作站直連 99 SSH 缺 public-key auth 時,`collect-windows99-vmware-verify.sh` 必須支援 `--via-host wooo@192.168.0.110`,沿 110→99 已驗證 no-secret path 執行 Verify;collector 必須輸出 `via_host=`,仍固定 BatchMode/publickey、in-memory stdin scriptblock、`remote_write_performed=false`、`vm_power_change_performed=false`。`validate-windows99-console-verify-artifact.py` default required aliases 必須與 v1.118 一致為 `110 / 188 / 120 / 121 / 112`;若未來要驗 explicit `111` artifact,必須用 `--required-aliases 111,188,120,121,112` 顯式指定。2026-07-03 via-host live artifact 已證明新版 Verify readback:VMX missing 為空、config ready、Windows Update no-auto-reboot ready,剩餘 blocker 是 `windows99_vmware_guest_power_not_ready`;不得再把 validator 舊預設造成的 `111` missing 當真 blocker。
|
||||
|
||||
2026-07-02 110 control-path / Harbor recovery receipt rule:若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure,但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata,不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier,並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示:public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`;active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure,以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker,non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker;non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。
|
||||
|
||||
2026-07-02 110 controlled CD lane fail-closed enforcer staging rule:110 runner 壓力事故後,legacy / generic runner 仍必須 fail-closed;但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail,且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit,並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service;若 registration 缺失,readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail,不要手工反覆補同一組 artifact。
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# AWOOOI 重開機恢復 SOP
|
||||
|
||||
> **版本**: v5.10
|
||||
> **版本**: v5.11
|
||||
> **最後更新**: 2026-07-03 (台北時間)
|
||||
> **更新者**: Codex
|
||||
> **觸發事件**: 2026-07-03 13:40 deploy 後 source-of-truth reconciliation:99 VMware default VM target 回到 runtime apply truth `110 / 188 / 120 / 121 / 112`;111 維持 required host / explicit VMX override,不再無證據當作 99 預設 VMX blocker
|
||||
> **觸發事件**: 2026-07-03 13:50 no-secret via-host Verify readback:新版 99 verifier 已證明 default VMX source 不缺,剩餘 blocker 是 VM power/readback;validator / collector source 已同步
|
||||
|
||||
---
|
||||
|
||||
@@ -52,6 +52,8 @@
|
||||
|
||||
2026-07-03 13:40 source-of-truth overlay:7/1 runtime apply evidence 顯示 99 實際 VMware layout 是 `110 -> 188 -> 120 -> 121 -> 112`,且當時已標記 `host111_vmware_target=missing_no_vmx_found_on_99_ping_only`。因此 repo source、API scorecard fallback、CLI scorecard fallback 必須使用 `110 / 188 / 120 / 121 / 112` 作為 Windows99 default VM target;`111` 仍是 reboot SLO required host,但不是 99 VMware 預設 VM。若未來取得 verified `Host111Vmx` 或 verifier stdout 明確列出 `VMX alias=111` / `VM_POWER alias=111`,scorecard 仍會用 verifier 實際 alias 集合把 `111` 納入 repair package。禁止用 generic Ubuntu candidate 推測成 111、禁止為了清 blocker 修改 Windows task、relink、restore、啟動 VM 或重啟 host。
|
||||
|
||||
2026-07-03 13:50 no-secret verifier overlay:本工作站直連 99 仍回 `blocked_ssh_publickey_auth_missing`,但 `collect-windows99-vmware-verify.sh --collect --via-host wooo@192.168.0.110 --users wooo --max-auth-users 1` 成功取得 Verify stdout。validator default 已改成 `110 / 188 / 120 / 121 / 112`,同輪 artifact 驗證為 `valid_console_verify_artifact_vmware_not_ready`、`missing_vmx_aliases=[]`、`powered_off_aliases=110,188,120,121,112`、`VMWARE_AUTOSTART_CONFIG_READY=1`、`WINDOWS_UPDATE_NO_AUTO_REBOOT_READY=1`、`VMWARE_AUTOSTART_VERIFY_READY=0`。因此目前 Windows99 lane 不再是 111 VMX source repair;下一步是部署新版 SLO exporter/collector 到 110、重跑 verifier service,讓 Prometheus / production scorecard 清掉舊 `windows99_vmware_vmx_missing`,再處理 VM power/readback blocker。仍不得啟動 VM、relink、restore、重啟 host 或讀密碼。
|
||||
|
||||
### 五主機全貌
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user