fix(reboot): gate windows99 vmx candidate confirmation
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m25s
CD Pipeline / build-and-deploy (push) Successful in 5m23s
CD Pipeline / post-deploy-checks (push) Successful in 1m47s
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m25s
CD Pipeline / build-and-deploy (push) Successful in 5m23s
CD Pipeline / post-deploy-checks (push) Successful in 1m47s
This commit is contained in:
@@ -1,11 +1,12 @@
|
||||
## 2026-07-03 — 13:12 Windows99 VMX relink dry-run evidence gate 上卷
|
||||
## 2026-07-03 — 13:12 Windows99 VMX relink dry-run / identity gate 上卷
|
||||
|
||||
**完成內容**:
|
||||
- Gitea CD `#4582` 已完成最新 main `ba0849f70` 部署,deploy marker 為 `558b9e916 chore(cd): deploy ba0849f [skip ci]`;job3 讀回 `ArgoCD: sync=Synced health=Healthy`、API/Web/Worker rollout completed、`Production deploy readback matches this build`、`Job succeeded`。
|
||||
- Production `/api/v1/agents/windows99-vmx-source-locator-readback` 已讀回 collector receipt:`collector_readback_present=true`、`candidate_source_count=6`、`identity_unassigned_ubuntu_candidate_count=1`、`raw_path_output=false`、`path_fingerprint_only=true`、`remote_write_performed=false`、`vm_power_change_performed=false`。
|
||||
- 本輪 verifier-only artifact:`/tmp/awoooi-p0-006-reboot-slo-20260703-130633`。host probe 顯示 99 reachable-unknown-boot、111 unreachable、112/120/121 reachable、188 reachable 但 `awoooi-startup.service failed/degraded`;本工作機對 99 的 no-secret SSH collector 固定 `ssh_batchmode_auth_ready=0`、`verify_collection_status=blocked_ssh_publickey_auth_missing`。
|
||||
- 將 VMX locator 的「confirmation」字眼改成 machine-readable evidence gate:`verified_source_or_identity_evidence_required=true`、`human_confirmation_required_for_dry_run=false`、`controlled_relink_dry_run_ready=false`、`controlled_relink_dry_run_blocker=verified_vmx_source_or_identity_evidence_missing`,並新增 `controlled_relink_dry_run_plan`。此 plan 只允許 dry-run evidence,不允許 apply / remote write / VM power change。
|
||||
- `awoooi-priority-work-order-readback` 同步投影上述 dry-run blocker 到 `mainline_execution_state`、in-progress evidence、`summary` 與 `rollups`,避免再把 P0-006 卡成模糊的人工確認。
|
||||
- 同步新增 `candidate_confirmation_gate`,schema 固定為 `windows99_vmx_source_candidate_confirmation_gate_v1`,把「唯一未指派 Ubuntu candidate」拆成 machine-readable identity gate;目前 `candidate_identity_evidence_sufficient=false`、`candidate_relink_dry_run_ready=false`、`apply_allowed_by_candidate_confirmation_gate=false`。
|
||||
- `awoooi-priority-work-order-readback` 同步投影上述 dry-run blocker 與 identity gate 到 `mainline_execution_state`、in-progress evidence、`summary` 與 `rollups`;AwoooP Work Items 補上 identity gate、dry-run gate、missing evidence 與 apply gate chips,避免再把 P0-006 卡成模糊人工確認。
|
||||
|
||||
**已跑驗證**:
|
||||
- `python3.11 -m py_compile apps/api/src/services/windows99_vmx_source_locator_readback.py apps/api/src/services/awoooi_priority_work_order_readback.py apps/api/tests/test_windows99_vmx_source_locator_readback_api.py apps/api/tests/test_awoooi_priority_work_order_readback_api.py`:通過。
|
||||
|
||||
@@ -64,6 +64,8 @@ v1.115 Windows99 VMX source locate check-mode rule:2026-07-03 12:34 deploy 後
|
||||
|
||||
v1.116 Windows99 VMX locator via-host execution rule:若目前工作站直連 99 缺 public-key auth,但 110 到 99 的 no-secret SSH path 已驗證,官方 locator 必須支援 `--via-host`,固定可執行命令為 `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect --via-host wooo@192.168.0.110 --users wooo --max-auth-users 1`。2026-07-03 live readback:`locator_collection_status=collected_windows99_vmx_source_locator_stdout`、`expected_vmx_path_present_count=0`、`candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`、`safe_next_step=provide_existing_vmx_source_or_backup_path_then_rerun_locator_check_mode_no_write`,且 `secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`、`raw_path_output=false`、`path_fingerprint_only=true`。此結果把下一步收斂為「提供既有 VMX source 或 verified backup path」;仍不得 relink generic VM、啟動 VM、restore 到 production 或讀密碼。
|
||||
|
||||
v1.117 Windows99 VMX candidate confirmation gate rule:locator receipt 顯示 `single_identity_unassigned_ubuntu_candidate_requires_confirmation` 時,不能把「唯一 generic Ubuntu candidate」直接升級成 111 VMX source。`/api/v1/agents/windows99-vmx-source-locator-readback` 必須輸出 `candidate_confirmation_gate`,並同步到 priority readback / Work Items;目前 gate 固定 `candidate_identity_evidence_sufficient=false`、`candidate_relink_dry_run_ready=false`、`apply_allowed_by_candidate_confirmation_gate=false`,missing evidence 至少包含 `target_alias_identity_hint`、`verified_backup_or_authorized_console_candidate_identity`、`pre_apply_backup_plan`、`scoped_relink_dry_run`。只有取得 target alias / MAC / UUID / display-name 或 verified backup path 證據,且產生 rollback + post-verifier 的 scoped relink dry-run 後,才可進下一階段;scorecard / locator / work item 均不得授權 VM power change、Windows write、service restart、host reboot、secret/password 讀取或 production restore。
|
||||
|
||||
2026-07-02 110 control-path / Harbor recovery receipt rule:若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure,但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata,不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier,並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示:public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`;active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure,以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker,non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker;non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。
|
||||
|
||||
2026-07-02 110 controlled CD lane fail-closed enforcer staging rule:110 runner 壓力事故後,legacy / generic runner 仍必須 fail-closed;但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail,且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit,並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service;若 registration 缺失,readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail,不要手工反覆補同一組 artifact。
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# AWOOOI 重開機恢復 SOP
|
||||
|
||||
> **版本**: v5.8
|
||||
> **版本**: v5.9
|
||||
> **最後更新**: 2026-07-03 (台北時間)
|
||||
> **更新者**: Codex
|
||||
> **觸發事件**: 2026-07-03 12:34 deploy 後 production SLO scorecard + Windows99 VMX source locator readback:111 VMX source 尚未找到,10 分鐘 SLO 仍 blocked
|
||||
> **觸發事件**: 2026-07-03 13:10 deploy 後 production SLO scorecard + Windows99 VMX source locator confirmation gate:111 VMX candidate 尚未取得足夠 identity evidence,10 分鐘 SLO 仍 blocked
|
||||
|
||||
---
|
||||
|
||||
@@ -48,6 +48,8 @@
|
||||
|
||||
固定下一步:`locate_or_relink_missing_vmx_source_check_mode`。production `/api/v1/agents/windows99-vmx-source-locator-readback` 已回 `status=check_mode_locator_ready_windows99_vmx_source_required`、`missing_vmx_aliases=["111"]`、`safe_next_step=bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect`、`secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_allowed=false`。目前工作站必須加 `--via-host wooo@192.168.0.110 --users wooo --max-auth-users 1` 走已驗證的 110 no-secret path;live 結果為 `locator_collection_status=collected_windows99_vmx_source_locator_stdout`、`expected_vmx_path_present_count=0`、`candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`、`raw_path_output=false`、`path_fingerprint_only=true`。下一步只能取得既有 VMX source 或 external verified backup path,再產生 scoped relink / restore dry-run 與 post-verifier。若 public API response 顯示 `public_lan_topology_redacted=true` 或 `check_mode_probe_commands_executable=false`,其中 `host:...` VMX path 只能當 public display evidence,不得直接貼到 Windows console 執行;要執行 probe 時必須使用授權內部 console / no-secret package 保留的原始 Windows path。禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。
|
||||
|
||||
2026-07-03 13:10 最新 overlay:Gitea CD `#4582` 已推 deploy marker `558b9e9`,source `ba0849f70`,production deploy readback matched=true 且 job_succeeded=true;Gitea UI 若仍顯示 Running,只能視為 run-level lag。production `/api/v1/agents/windows99-vmx-source-locator-readback` 已上卷 redacted locator receipt:`candidate_source_count=6`、`alias_hint_candidate_count=0`、`unassigned_ubuntu_candidate_count=5`、`identity_unassigned_ubuntu_candidate_count=1`、`candidate_confirmation_required=true`,狀態為 `collector_readback_single_identity_candidate_confirmation_required`。新的 `candidate_confirmation_gate` 固定輸出 `candidate_identity_evidence_sufficient=false`、`candidate_relink_dry_run_ready=false`、`apply_allowed_by_candidate_confirmation_gate=false`;缺少 evidence 為 `target_alias_identity_hint`、`verified_backup_or_authorized_console_candidate_identity`、`pre_apply_backup_plan`、`scoped_relink_dry_run`。因此下一步已從「重跑 locator」收斂為:取得授權 console candidate identity evidence 或 verified backup path,重跑 confirmation gate,再產生 scoped relink dry-run + rollback + post-verifier;仍不得猜測 generic Ubuntu candidate、不得 relink、不得 restore、不得 power on VM。
|
||||
|
||||
### 五主機全貌
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user