diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 141a935fa..cd806efc0 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,23 @@ +## 2026-07-03 — 17:18 Windows99 VMX repair CLI 冷啟動修正 + +**完成內容**： +- Gitea CD `#4617` 已完成 source `40713b2f8` 投產：deploy marker `b85d574`，production deploy readback matched=true，job succeeded；ArgoCD rollout 與 API health 均通過。 +- Production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已讀回新版 Windows99 truth：`required_vm_aliases=["111","188","120","121","112"]`、`missing_vmx_aliases=["111"]`、`powered_off_aliases=["112","120","121","188","111"]`，舊 `110` 不再出現在 Windows99 VM missing / powered-off alias。 +- via-110 no-secret VMX locator 本輪未進入 remote locator：`locator_collection_status=blocked_ssh_publickey_auth_missing`、`ssh_auth_probe_exit_status=124`，且 `secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`；這是 management-channel / locator blocker，不是 111 已修復。 +- 修正 `build-windows99-vmx-source-repair-package.py`：CLI 在冷啟動 / 無 DB / 無 `.env` 場景下會自帶 harmless placeholder `DATABASE_URL` 供 API builder import 驗證，避免 no-secret repair package 因本機環境缺 `DATABASE_URL` 假性失敗。 +- production scorecard repair package 目前可直接產生 `windows99-vmx-source-repair-111`：`status=check_mode_package_ready_windows99_vmx_source_repair_required`、`package_ready=true`、`missing_vmx_aliases=["111"]`、`apply_allowed_by_this_package=false`、`rollback_required=true`、`vm_power_change_allowed=false`。 + +**已跑驗證**： +- `python3.11 -m pytest scripts/reboot-recovery/tests/test_windows99_vmx_source_repair_package.py -q -p no:cacheprovider`：`4 passed`。 +- `python3.11 scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py --timeout-seconds 12`：成功輸出 `windows99_vmx_source_repair_check_mode_package_v1`，且 operation boundaries 全部為 no-secret / no-write / no-VM-power-change。 + +**仍維持**： +- 這不代表 10 分鐘重啟自動恢復 SLO 完成；目前 primary blocker 仍是 `reboot_event_required_host_unreachable`，Windows99 lane 仍 blocked by `111` VMX missing、guest powered off、scheduled task never ran。 +- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth；沒有使用 GitHub / gh；沒有啟動或關閉 VM，沒有重啟 Windows host / VMware service，沒有 Docker / Nginx / K3s / DB / firewall restart / reload，沒有 DROP / TRUNCATE / restore / prune / delete / force push。 + +**下一步**： +- commit / push 到 Gitea main；CD 後讀回 production package CLI / endpoint。P0-4 下一步固定為取得 111 既有 VMX source / verified backup / 授權 console candidate identity，再產生 scoped relink dry-run + rollback + post-verifier；不得猜 generic Ubuntu candidate 或直接 power on VM。 + ## 2026-07-03 — 16:33 Windows99 VMware default target 修正為 111 **完成內容**： diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 2b856304b..1441546fe 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,6 +1,6 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.125 +> Version: v1.127 > Last updated: 2026-07-03 Asia/Taipei > Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號，但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal；此納入不代表授權任何破壞性 runtime apply。 @@ -84,6 +84,8 @@ v1.125 Windows99 autostart task run-state rule：`windows99-vmware-autostart.ps1 v1.126 Windows99 bounded verifier / fallback compatibility rule：`windows99-vmware-autostart.ps1 -Mode Verify` 預設不得遞迴掃描 VM roots；111 VMX 缺失時要快速輸出 `VMX alias=111 present=0` 與 `VM_POWER alias=111 vmx_present=0 running=0`，深度尋找 VMX 必須走獨立 locator / check-mode lane。`reboot-auto-recovery-slo-exporter.sh` 的 Windows99 fallback artifact 必須與目前 default alias set `111 / 188 / 120 / 121 / 112` 完全相容；若舊 artifact 含 `110` 或缺 `111`，不得回灌 textfile / scorecard。這是避免 timeout 後用舊 110 artifact 假綠的防線；不授權 power change、relink、restore、host reboot 或 secret 讀取。 +v1.127 Windows99 VMX repair CLI cold-start rule：`scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py` 是 no-secret / no-DB / check-mode recovery tool，不得因工作站沒有 runtime `DATABASE_URL` 或 `.env` 就失敗；CLI 必須可用 harmless placeholder DB URL 完成 import 與 package build，但不得連 DB、不得讀 secret、不得寫 Windows、不得 power change。2026-07-03 17:18 production scorecard package readback：`work_order_id=windows99-vmx-source-repair-111`、`missing_vmx_aliases=["111"]`、`apply_allowed_by_this_package=false`、`rollback_required=true`、`vm_power_change_allowed=false`。同輪 via-110 locator 若回 `blocked_ssh_publickey_auth_missing` / `ssh_auth_probe_exit_status=124`，只能當 management-channel blocker；不得把 repair package ready 誤報成 111 VMX 已恢復。 + 2026-07-02 110 control-path / Harbor recovery receipt rule：若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure，但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`，則該 Gitea Harbor repair 失敗只能列為 historical queue metadata，不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier，並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示：public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`；active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure，以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker，non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker；non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。 2026-07-02 110 controlled CD lane fail-closed enforcer staging rule：110 runner 壓力事故後，legacy / generic runner 仍必須 fail-closed；但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail，且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時，才可保留 drain config / binary / unit，並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service；若 registration 缺失，readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走，優先修 source enforcer / unit guardrail，不要手工反覆補同一組 artifact。 diff --git a/docs/runbooks/REBOOT-RECOVERY-SOP.md b/docs/runbooks/REBOOT-RECOVERY-SOP.md index d87e6591f..3ee1ec347 100644 --- a/docs/runbooks/REBOOT-RECOVERY-SOP.md +++ b/docs/runbooks/REBOOT-RECOVERY-SOP.md @@ -1,9 +1,9 @@ # AWOOOI 重開機恢復 SOP -> **版本**: v5.17 +> **版本**: v5.18 > **最後更新**: 2026-07-03 (台北時間) > **更新者**: Codex -> **觸發事件**: 2026-07-03 15:57 Windows99 VMware autostart task exists/enabled 但 `0x41303` 顯示未觸發，五台 VM guest 仍 powered off +> **觸發事件**: 2026-07-03 17:18 Windows99 VMX source repair package CLI 冷啟動修正，111 VMX missing 轉入 check-mode repair lane --- @@ -21,6 +21,12 @@ ## 架構概覽與依賴圖 +### 2026-07-03 17:18 P0-4 VMX Repair CLI 覆蓋 + +Gitea CD `#4617` 已完成 source `40713b2f8` 投產，deploy marker `b85d574`，production deploy readback matched=true，job succeeded。production reboot SLO scorecard 已讀回 Windows99 default target `111 / 188 / 120 / 121 / 112`，其中 `missing_vmx_aliases=["111"]`、`powered_off_aliases=["112","120","121","188","111"]`，舊 `110` 不得再當 Windows99 guest alias。 + +`build-windows99-vmx-source-repair-package.py` 是 no-secret / no-DB / check-mode 工具，必須能在工作站缺 `DATABASE_URL` 時輸出 `windows99-vmx-source-repair-111` package；這只表示 repair check-mode package ready，不表示 111 VMX 已恢復。via-110 locator 若回 `blocked_ssh_publickey_auth_missing` 或 `ssh_auth_probe_exit_status=124`，固定視為 management-channel blocker。下一步是取得 111 既有 VMX source / verified backup / 授權 console candidate identity，建立 scoped relink dry-run、rollback 與 post-verifier；不得猜 generic Ubuntu candidate、不得直接 relink、不得 restore、不得 power on VM、不得讀 Windows 密碼。 + ### 2026-07-03 15:57 P0-4 Windows99 Task Run-State 覆蓋 Windows99 no-secret via-110 collector 已讀回 `AWOOOI-Start-VMware-VMs` scheduled task：`present=1`、`enabled=1`、`state=Ready`、`trigger_count=1`、`action_count=1`，VMware service 與 Windows Update no-auto-reboot policy 也為 ready；但新增 task detail 顯示 `last_run_time=1999-11-30T00:00:00`、`last_task_result=267011`，等價於 `0x41303`，表示 task 尚未跑過 / 未在本輪 boot 觸發。2026-07-03 16:33 進一步確認舊 verifier default 把 `110` 誤列為 Windows99 guest；最新 source-of-truth 改為 `111 / 188 / 120 / 121 / 112`，以對齊本輪 host probe `111 unreachable` 與 99 VMware guest 需求。因此目前 P0-4 真 blocker 是 `windows99_vmware_guest_power_not_ready` 加 `windows99_vmware_autostart_task_never_ran`，且 deployment 後必須用新版 collector 重新讀回 111。 diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index 74ad3e76a..e856fac21 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -22,7 +22,7 @@ | P0-1 | BLOCKED_REBOOT_EVENT_AND_HOST_REACHABILITY | 全主機 reboot auto-detection / auto-trigger / 10 分鐘恢復 SLO | 2026-07-03 15:49 verify-only artifact `/tmp/awoooi-reboot-event-verify-20260703-154907`：`reboot_detected=false`、`all_required_hosts_observed=false`、`all_required_hosts_in_reboot_window=false`、`recovery_deadline_status=target_window_elapsed`、`unreachable_hosts=["111"]`、99 reachable 但 `uptime_seconds=unknown`、188 reachable 但 `systemd_state=degraded` / `startup_active=failed`。production SLO 仍為 `blocked_reboot_auto_recovery_slo_not_ready`，不能宣稱 10 分鐘自動恢復。 | 固定先 rerun host probe + reboot-event detector verify-only，並收斂 111 reachability、99 uptime / VMware readback、188 degraded/startup failed；不得 reboot、不得 VM power change、不得讀 Windows 密碼，不得用 ping-only 或舊 boot evidence 當完成證據。 | | P0-2 | DEPLOYED_RUNTIME_EDGE_DRIFT_BLOCKED | Deploy / reboot 期間 public 502 維護頁與外部 fallback | 2026-07-03 15:35 source fix 已部署讀回：production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 保留 `edge_public_maintenance_live_config_drift`，`runtime_metric_active_blocker_count=8`，且 route fallback ready 不再清掉 edge Nginx drift。110 SLO artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-151219` 與 textfile 證明 edge 未 ready：`awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"} 0`、active blocker `edge_public_maintenance_live_config_drift`。 | 後續才進 privileged edge apply 或 L1 cloud/CDN fallback decision；完成條件是 edge fallback live config / maintenance file / `nginx -t` / reload / public header verifier 或外部 L1 fallback 全綠。不得讀密碼、不得用 app / Docker / host restart 掩蓋 edge fallback drift。 | | P0-3 | PARTIAL_GREEN_DEPLOYED_SOURCE | 所有產品 / 網站版本與資料最新性 | Gitea `main=bc8d3af3f chore(cd): deploy dacdd90 [skip ci]`；CD `#4558` log 顯示 API/Web build/push wrapper 全部 `rc=0`、deploy marker `dacdd90`、production deploy readback matched。Production SLO route 已讀回最新 API 行為：`runtime_metric_runtime_readback_added_blockers=["windows99_vmware_autostart_config_not_ready","windows99_update_no_auto_reboot_policy_not_ready"]`、`windows99_update_no_auto_reboot_ready=false`。Stock freshness 仍為 `status=ok`、`latest_trading_date=2026-07-02`、blockers `[]`。 | 版本層已補到 AWOOOI production；完成整體 P0 仍需收斂 Windows99 VMX / service / policy / power 與 host reboot-event blockers。後續每個 public product 仍要維持 source SHA、deploy marker、runtime endpoint、public route watch、freshness 四層一致。 | -| P0-4 | BLOCKED_WINDOWS99_TASK_NOT_TRIGGERED | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | 2026-07-03 16:33 production readback 已證明 source/API/textfile 都會輸出 `windows99_vmware_autostart_task_never_ran`，但同時發現舊 source-of-truth 把 Windows99 default VM target 寫成 `110 / 188 / 120 / 121 / 112`，與本輪事故需求及 host probe `111 unreachable` 不一致。最新 source 修正為 `111 / 188 / 120 / 121 / 112`，110 回到 P0 host / control-plane truth，不再當 99 VMware guest default。 | 部署修正後重新跑 no-secret via-110 collector + SLO scorecard，確認 `required_vm_aliases`、`powered_off_aliases`、check-mode package、validator 與 API readback 都以 111 為 Windows99 VM target。仍不得從 scorecard 直接 power on VM、重啟 Windows host / VMware service、改 scheduled task / registry 或讀 Windows 密碼。 | +| P0-4 | BLOCKED_WINDOWS99_VMX_SOURCE_AND_TASK_NOT_TRIGGERED | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | 2026-07-03 17:18 CD `#4617` 已完成 source `40713b2f8` 投產，deploy marker `b85d574`，production deploy readback matched=true，job succeeded。production scorecard 已讀回 `required_vm_aliases=["111","188","120","121","112"]`、`missing_vmx_aliases=["111"]`、`powered_off_aliases=["112","120","121","188","111"]`，舊 `110` 不再出現在 Windows99 VM alias。`build-windows99-vmx-source-repair-package.py` 已修正可在無 `DATABASE_URL` 冷啟動環境產生 `windows99-vmx-source-repair-111` package；via-110 locator 本輪仍回 `blocked_ssh_publickey_auth_missing` / `ssh_auth_probe_exit_status=124`。 | 下一步固定為取得 111 既有 VMX source / verified backup / 授權 console candidate identity，產生 scoped relink dry-run + rollback + post-verifier，再 rerun no-secret collector + SLO scorecard。仍不得從 scorecard 直接 power on VM、重啟 Windows host / VMware service、改 scheduled task / registry、猜 generic Ubuntu candidate、restore 到 production 或讀 Windows 密碼。 | | P0-5 | RUNTIME_READY_BACKUP_RECEIPT_GAP | Gitea / 主機 / DB / 網站 / 服務 / 套件 / 工具 / log 備份監控告警 | Gitea repo bundle readback ready：expected `12`、rows `12`、missing `0`、failed `0`、sample restore dry-run ok；backup core green。2026-07-03 source / runtime 已部署 `awoooi_backup_alert_receipt_*` 指標與 Prometheus rules；110 exporter 讀回 88 個 stage requirement、188 讀回 12 個 stage requirement，`BackupAlertReceiptMetricMissing*` inactive，`BackupAlertReceiptStageMissing` 已修成每 `host / receipt_channel` 聚合 pending：110 一條、188 一條。 | 補 `/backup/alert-receipts/*.last_success` 脫敏 marker；下一層仍要補 Gitea full dump、DB/settings/issues/packages/LFS、所有工具與 log 全量備份監控。 | | P0-6 | RUNTIME_READY_ALERT_RECEIPT_GAP | 主機關機 / 重啟 / SLO miss / backup failure Telegram 告警 | Reboot per-blocker alert 與 backup receipt alert rules 已 deploy/readback；backup receipt 缺段不再產生 100 條 stage 噪音，現在聚合成 110 / 188 兩條 host-level pending。scorecard 目前有 10 個 reboot active blockers，其中 Windows99 config / policy blocker 已可供 per-blocker alert routing 使用；尚未完成 shutdown / reboot / backup alert 的 production 脫敏 delivery receipt 全矩陣。 | 補 alert receipt readback：host down、host up、SLO miss、Windows99 blocker、backup stale/failed、deploy 502、freshness stale；完成條件是每類告警都有 sent / received / dedup / escalation evidence。 | | P0-7 | SOURCE_READY_SLA_AUTOMATION | 固定排查順序、ETA / wait reason、自動化判斷與修復 | Scorecard 已固定 `current_phase`、`eta_or_wait_reason`、`primary_blocker`、fixed triage order 與 action matrix；2026-07-03 新增 `windows99_vmware_autostart_task_never_ran`，其 next action 固定為 `run_windows99_vmware_autostart_check_mode_package_for_powered_off_aliases_then_rerun_no_secret_collector_no_vm_power_change`，不再退回泛用「再收一次 verify」。 | 把每個 blocker 的 next_safe_action、post_verifier、forbidden_actions 接到自動 work item / Telegram / scorecard；完成條件是重啟後自動判斷、主動告警、主動 rerun verifier，不再人工臨場猜流程。 | diff --git a/scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py b/scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py index fcb24714d..17742ab40 100644 --- a/scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py +++ b/scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py @@ -10,6 +10,7 @@ from __future__ import annotations import argparse import json +import os import sys import urllib.request from pathlib import Path @@ -20,6 +21,11 @@ _API_ROOT = _REPO_ROOT / "apps" / "api" if str(_API_ROOT) not in sys.path: sys.path.insert(0, str(_API_ROOT)) +# This CLI is a no-DB, check-mode reboot recovery tool. The API package import +# still validates settings, so provide a harmless placeholder when running +# outside the service container. +os.environ.setdefault("DATABASE_URL", "postgresql+asyncpg://test:test@localhost/test") + from src.services.reboot_auto_recovery_slo_scorecard import ( # noqa: E402 _build_windows99_vmx_source_repair_package, _taipei_now_iso, diff --git a/scripts/reboot-recovery/tests/test_windows99_vmx_source_repair_package.py b/scripts/reboot-recovery/tests/test_windows99_vmx_source_repair_package.py index 7ae760286..799b42801 100644 --- a/scripts/reboot-recovery/tests/test_windows99_vmx_source_repair_package.py +++ b/scripts/reboot-recovery/tests/test_windows99_vmx_source_repair_package.py @@ -2,6 +2,7 @@ from __future__ import annotations import importlib.util import json +import os import subprocess from pathlib import Path @@ -172,6 +173,7 @@ def test_cli_writes_package_file(tmp_path: Path) -> None: capture_output=True, text=True, check=False, + env={key: value for key, value in os.environ.items() if key != "DATABASE_URL"}, ) assert result.returncode == 0