fix(reboot): gate windows99 vmx candidate confirmation
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m25s
CD Pipeline / build-and-deploy (push) Successful in 5m23s
CD Pipeline / post-deploy-checks (push) Successful in 1m47s

This commit is contained in:
Your Name
2026-07-03 13:13:51 +08:00
parent bcaf6c3a26
commit 232d680962
10 changed files with 471 additions and 4 deletions

View File

@@ -1,11 +1,12 @@
## 2026-07-03 — 13:12 Windows99 VMX relink dry-run evidence gate 上卷
## 2026-07-03 — 13:12 Windows99 VMX relink dry-run / identity gate 上卷
**完成內容**
- Gitea CD `#4582` 已完成最新 main `ba0849f70` 部署deploy marker 為 `558b9e916 chore(cd): deploy ba0849f [skip ci]`job3 讀回 `ArgoCD: sync=Synced health=Healthy`、API/Web/Worker rollout completed、`Production deploy readback matches this build``Job succeeded`
- Production `/api/v1/agents/windows99-vmx-source-locator-readback` 已讀回 collector receipt`collector_readback_present=true``candidate_source_count=6``identity_unassigned_ubuntu_candidate_count=1``raw_path_output=false``path_fingerprint_only=true``remote_write_performed=false``vm_power_change_performed=false`
- 本輪 verifier-only artifact`/tmp/awoooi-p0-006-reboot-slo-20260703-130633`。host probe 顯示 99 reachable-unknown-boot、111 unreachable、112/120/121 reachable、188 reachable 但 `awoooi-startup.service failed/degraded`;本工作機對 99 的 no-secret SSH collector 固定 `ssh_batchmode_auth_ready=0``verify_collection_status=blocked_ssh_publickey_auth_missing`
- 將 VMX locator 的「confirmation」字眼改成 machine-readable evidence gate`verified_source_or_identity_evidence_required=true``human_confirmation_required_for_dry_run=false``controlled_relink_dry_run_ready=false``controlled_relink_dry_run_blocker=verified_vmx_source_or_identity_evidence_missing`,並新增 `controlled_relink_dry_run_plan`。此 plan 只允許 dry-run evidence不允許 apply / remote write / VM power change。
- `awoooi-priority-work-order-readback` 同步投影上述 dry-run blocker 到 `mainline_execution_state`、in-progress evidence、`summary``rollups`,避免再把 P0-006 卡成模糊的人工確認
- 同步新增 `candidate_confirmation_gate`schema 固定為 `windows99_vmx_source_candidate_confirmation_gate_v1`,把「唯一未指派 Ubuntu candidate」拆成 machine-readable identity gate目前 `candidate_identity_evidence_sufficient=false``candidate_relink_dry_run_ready=false``apply_allowed_by_candidate_confirmation_gate=false`
- `awoooi-priority-work-order-readback` 同步投影上述 dry-run blocker 與 identity gate 到 `mainline_execution_state`、in-progress evidence、`summary``rollups`AwoooP Work Items 補上 identity gate、dry-run gate、missing evidence 與 apply gate chips避免再把 P0-006 卡成模糊人工確認。
**已跑驗證**
- `python3.11 -m py_compile apps/api/src/services/windows99_vmx_source_locator_readback.py apps/api/src/services/awoooi_priority_work_order_readback.py apps/api/tests/test_windows99_vmx_source_locator_readback_api.py apps/api/tests/test_awoooi_priority_work_order_readback_api.py`:通過。

View File

@@ -64,6 +64,8 @@ v1.115 Windows99 VMX source locate check-mode rule2026-07-03 12:34 deploy 後
v1.116 Windows99 VMX locator via-host execution rule若目前工作站直連 99 缺 public-key auth但 110 到 99 的 no-secret SSH path 已驗證,官方 locator 必須支援 `--via-host`,固定可執行命令為 `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect --via-host wooo@192.168.0.110 --users wooo --max-auth-users 1`。2026-07-03 live readback`locator_collection_status=collected_windows99_vmx_source_locator_stdout``expected_vmx_path_present_count=0``candidate_source_count=0``source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found``safe_next_step=provide_existing_vmx_source_or_backup_path_then_rerun_locator_check_mode_no_write`,且 `secret_value_read=false``remote_write_performed=false``vm_power_change_performed=false``raw_path_output=false``path_fingerprint_only=true`。此結果把下一步收斂為「提供既有 VMX source 或 verified backup path」仍不得 relink generic VM、啟動 VM、restore 到 production 或讀密碼。
v1.117 Windows99 VMX candidate confirmation gate rulelocator receipt 顯示 `single_identity_unassigned_ubuntu_candidate_requires_confirmation` 時,不能把「唯一 generic Ubuntu candidate」直接升級成 111 VMX source。`/api/v1/agents/windows99-vmx-source-locator-readback` 必須輸出 `candidate_confirmation_gate`,並同步到 priority readback / Work Items目前 gate 固定 `candidate_identity_evidence_sufficient=false``candidate_relink_dry_run_ready=false``apply_allowed_by_candidate_confirmation_gate=false`missing evidence 至少包含 `target_alias_identity_hint``verified_backup_or_authorized_console_candidate_identity``pre_apply_backup_plan``scoped_relink_dry_run`。只有取得 target alias / MAC / UUID / display-name 或 verified backup path 證據,且產生 rollback + post-verifier 的 scoped relink dry-run 後才可進下一階段scorecard / locator / work item 均不得授權 VM power change、Windows write、service restart、host reboot、secret/password 讀取或 production restore。
2026-07-02 110 control-path / Harbor recovery receipt rule若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/``401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh``recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335``Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` markernon110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blockernon110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。
2026-07-02 110 controlled CD lane fail-closed enforcer staging rule110 runner 壓力事故後legacy / generic runner 仍必須 fail-closed`awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host``awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner``CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail且 service `inactive``MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service若 registration 缺失readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail不要手工反覆補同一組 artifact。

View File

@@ -1,9 +1,9 @@
# AWOOOI 重開機恢復 SOP
> **版本**: v5.8
> **版本**: v5.9
> **最後更新**: 2026-07-03 (台北時間)
> **更新者**: Codex
> **觸發事件**: 2026-07-03 12:34 deploy 後 production SLO scorecard + Windows99 VMX source locator readback111 VMX source 尚未找到10 分鐘 SLO 仍 blocked
> **觸發事件**: 2026-07-03 13:10 deploy 後 production SLO scorecard + Windows99 VMX source locator confirmation gate111 VMX candidate 尚未取得足夠 identity evidence10 分鐘 SLO 仍 blocked
---
@@ -48,6 +48,8 @@
固定下一步:`locate_or_relink_missing_vmx_source_check_mode`。production `/api/v1/agents/windows99-vmx-source-locator-readback` 已回 `status=check_mode_locator_ready_windows99_vmx_source_required``missing_vmx_aliases=["111"]``safe_next_step=bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect``secret_value_read=false``remote_write_performed=false``vm_power_change_allowed=false`。目前工作站必須加 `--via-host wooo@192.168.0.110 --users wooo --max-auth-users 1` 走已驗證的 110 no-secret pathlive 結果為 `locator_collection_status=collected_windows99_vmx_source_locator_stdout``expected_vmx_path_present_count=0``candidate_source_count=0``source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found``raw_path_output=false``path_fingerprint_only=true`。下一步只能取得既有 VMX source 或 external verified backup path再產生 scoped relink / restore dry-run 與 post-verifier。若 public API response 顯示 `public_lan_topology_redacted=true``check_mode_probe_commands_executable=false`,其中 `host:...` VMX path 只能當 public display evidence不得直接貼到 Windows console 執行;要執行 probe 時必須使用授權內部 console / no-secret package 保留的原始 Windows path。禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。
2026-07-03 13:10 最新 overlayGitea CD `#4582` 已推 deploy marker `558b9e9`source `ba0849f70`production deploy readback matched=true 且 job_succeeded=trueGitea UI 若仍顯示 Running只能視為 run-level lag。production `/api/v1/agents/windows99-vmx-source-locator-readback` 已上卷 redacted locator receipt`candidate_source_count=6``alias_hint_candidate_count=0``unassigned_ubuntu_candidate_count=5``identity_unassigned_ubuntu_candidate_count=1``candidate_confirmation_required=true`,狀態為 `collector_readback_single_identity_candidate_confirmation_required`。新的 `candidate_confirmation_gate` 固定輸出 `candidate_identity_evidence_sufficient=false``candidate_relink_dry_run_ready=false``apply_allowed_by_candidate_confirmation_gate=false`;缺少 evidence 為 `target_alias_identity_hint``verified_backup_or_authorized_console_candidate_identity``pre_apply_backup_plan``scoped_relink_dry_run`。因此下一步已從「重跑 locator」收斂為取得授權 console candidate identity evidence 或 verified backup path重跑 confirmation gate再產生 scoped relink dry-run + rollback + post-verifier仍不得猜測 generic Ubuntu candidate、不得 relink、不得 restore、不得 power on VM。
### 五主機全貌
```