fix(reboot): surface windows99 console artifact blockers
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m13s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled

This commit is contained in:
Your Name
2026-07-03 09:25:20 +08:00
parent 43620a977f
commit e28ebd5b3e
10 changed files with 374 additions and 9 deletions

View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.107
> Version: v1.108
> Last updated: 2026-07-03 Asia/Taipei
> Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號,但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal此納入不代表授權任何破壞性 runtime apply。
@@ -46,6 +46,8 @@ v1.105 public maintenance edge source-to-live drift ruleGitea CD `#4519` 已
v1.107 2026-07-03 08:23 reboot verify-only readback ruleproduction `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 仍回 `status=blocked_reboot_auto_recovery_slo_not_ready``active_blocker_count=8``readiness_percent=67``primary_blocker=reboot_event_required_host_unreachable`active blockers 固定為 `all_required_hosts_not_in_10_minute_reboot_window``fresh_all_host_reboot_event_missing``host_boot_observation_older_than_target_window``host_unreachable_after_reboot``host_uptime_unknown``reboot_event_required_host_unreachable``windows99_vmware_guest_power_not_ready``windows99_vmware_vmx_missing`。同輪 no-write artifact `/tmp/awoooi-reboot-verify-only-20260703-082310` 證明99 ping/RDP/Hyper-V VMConnect 可達但 uptime unknownSSH BatchMode / WinRM 不可用111 仍 unreachable112/120/121 可達但 uptime 已超過 10 分鐘目標窗口188 可達但 `systemd_state=degraded``awoooi-startup.service failed`。因此不得宣稱 10 分鐘 SLO 已完成,也不得把 RDP 可見或 source verifier 存在當作 VMware autostart ready。下一步固定為 `restore_windows99_no_secret_management_channel_or_collect_local_console_verify_readback_then_rerun_reboot_scorecard_no_reboot`,並同時保持 `rerun_reboot_event_detector_and_host_probe_verify_only_no_reboot`;禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。
v1.108 Windows99 console artifact reliability blocker rule若 RDP / Hyper-V VMConnect 畫面可見,但 clipboard paste、視窗 focus 或 stdout 擷取不可靠,不能把畫面片段、截斷 output 或 Windows 端舊剪貼簿內容當成 `windows99-vmware-autostart.ps1 -Mode Verify` artifact。`windows99-management-channel-probe.py` 必須輸出 `console_artifact_status``console_artifact_reliable``console_artifact_blockers``console_artifact_safe_next_step``blocked_clipboard_unreliable``blocked_focus_unreliable``blocked_truncated_output` 分別轉成 `windows99_console_clipboard_unreliable``windows99_console_focus_unreliable``windows99_console_verify_output_truncated`。這些 blocker 必須進 `active_blockers``windows99_verify_collection.collection_blockers`、readback、rollups 與 `active_blocker_action_matrix`owner lane 固定為 `windows99_console_or_no_secret_management_channel`next safe action 固定為 `stop_unreliable_rdp_clipboard_path_and_use_authorized_no_secret_management_channel_or_validated_console_stdout_artifact`。完成條件仍是取得 validator 可接受的完整 no-secret Verify stdout 或恢復授權 no-secret management channel不得讀 Windows 密碼、不得啟動 / 關閉 VM、不得重啟 host / service。
2026-07-02 110 control-path / Harbor recovery receipt rule若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/``401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh``recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335``Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` markernon110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blockernon110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。
2026-07-02 110 controlled CD lane fail-closed enforcer staging rule110 runner 壓力事故後legacy / generic runner 仍必須 fail-closed`awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host``awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner``CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail且 service `inactive``MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service若 registration 缺失readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail不要手工反覆補同一組 artifact。