fix(reboot): keep vmx repair package cli no-db
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 1s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m12s
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 0s
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Successful in 16s
CD Pipeline / build-and-deploy (push) Successful in 4m13s
CD Pipeline / post-deploy-checks (push) Successful in 1m55s

This commit is contained in:
Your Name
2026-07-03 17:14:45 +08:00
parent b85d574383
commit 36536adcfa
6 changed files with 40 additions and 4 deletions

View File

@@ -1,3 +1,23 @@
## 2026-07-03 — 17:18 Windows99 VMX repair CLI 冷啟動修正
**完成內容**
- Gitea CD `#4617` 已完成 source `40713b2f8` 投產deploy marker `b85d574`production deploy readback matched=truejob succeededArgoCD rollout 與 API health 均通過。
- Production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已讀回新版 Windows99 truth`required_vm_aliases=["111","188","120","121","112"]``missing_vmx_aliases=["111"]``powered_off_aliases=["112","120","121","188","111"]`,舊 `110` 不再出現在 Windows99 VM missing / powered-off alias。
- via-110 no-secret VMX locator 本輪未進入 remote locator`locator_collection_status=blocked_ssh_publickey_auth_missing``ssh_auth_probe_exit_status=124`,且 `secret_value_read=false``remote_write_performed=false``vm_power_change_performed=false`;這是 management-channel / locator blocker不是 111 已修復。
- 修正 `build-windows99-vmx-source-repair-package.py`CLI 在冷啟動 / 無 DB / 無 `.env` 場景下會自帶 harmless placeholder `DATABASE_URL` 供 API builder import 驗證,避免 no-secret repair package 因本機環境缺 `DATABASE_URL` 假性失敗。
- production scorecard repair package 目前可直接產生 `windows99-vmx-source-repair-111``status=check_mode_package_ready_windows99_vmx_source_repair_required``package_ready=true``missing_vmx_aliases=["111"]``apply_allowed_by_this_package=false``rollback_required=true``vm_power_change_allowed=false`
**已跑驗證**
- `python3.11 -m pytest scripts/reboot-recovery/tests/test_windows99_vmx_source_repair_package.py -q -p no:cacheprovider``4 passed`
- `python3.11 scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py --timeout-seconds 12`:成功輸出 `windows99_vmx_source_repair_check_mode_package_v1`,且 operation boundaries 全部為 no-secret / no-write / no-VM-power-change。
**仍維持**
- 這不代表 10 分鐘重啟自動恢復 SLO 完成;目前 primary blocker 仍是 `reboot_event_required_host_unreachable`Windows99 lane 仍 blocked by `111` VMX missing、guest powered off、scheduled task never ran。
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth沒有使用 GitHub / gh沒有啟動或關閉 VM沒有重啟 Windows host / VMware service沒有 Docker / Nginx / K3s / DB / firewall restart / reload沒有 DROP / TRUNCATE / restore / prune / delete / force push。
**下一步**
- commit / push 到 Gitea mainCD 後讀回 production package CLI / endpoint。P0-4 下一步固定為取得 111 既有 VMX source / verified backup / 授權 console candidate identity再產生 scoped relink dry-run + rollback + post-verifier不得猜 generic Ubuntu candidate 或直接 power on VM。
## 2026-07-03 — 16:33 Windows99 VMware default target 修正為 111
**完成內容**

View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.125
> Version: v1.127
> Last updated: 2026-07-03 Asia/Taipei
> Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號,但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal此納入不代表授權任何破壞性 runtime apply。
@@ -84,6 +84,8 @@ v1.125 Windows99 autostart task run-state rule`windows99-vmware-autostart.ps1
v1.126 Windows99 bounded verifier / fallback compatibility rule`windows99-vmware-autostart.ps1 -Mode Verify` 預設不得遞迴掃描 VM roots111 VMX 缺失時要快速輸出 `VMX alias=111 present=0``VM_POWER alias=111 vmx_present=0 running=0`,深度尋找 VMX 必須走獨立 locator / check-mode lane。`reboot-auto-recovery-slo-exporter.sh` 的 Windows99 fallback artifact 必須與目前 default alias set `111 / 188 / 120 / 121 / 112` 完全相容;若舊 artifact 含 `110` 或缺 `111`,不得回灌 textfile / scorecard。這是避免 timeout 後用舊 110 artifact 假綠的防線;不授權 power change、relink、restore、host reboot 或 secret 讀取。
v1.127 Windows99 VMX repair CLI cold-start rule`scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py` 是 no-secret / no-DB / check-mode recovery tool不得因工作站沒有 runtime `DATABASE_URL``.env` 就失敗CLI 必須可用 harmless placeholder DB URL 完成 import 與 package build但不得連 DB、不得讀 secret、不得寫 Windows、不得 power change。2026-07-03 17:18 production scorecard package readback`work_order_id=windows99-vmx-source-repair-111``missing_vmx_aliases=["111"]``apply_allowed_by_this_package=false``rollback_required=true``vm_power_change_allowed=false`。同輪 via-110 locator 若回 `blocked_ssh_publickey_auth_missing` / `ssh_auth_probe_exit_status=124`,只能當 management-channel blocker不得把 repair package ready 誤報成 111 VMX 已恢復。
2026-07-02 110 control-path / Harbor recovery receipt rule若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/``401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh``recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335``Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` markernon110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blockernon110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。
2026-07-02 110 controlled CD lane fail-closed enforcer staging rule110 runner 壓力事故後legacy / generic runner 仍必須 fail-closed`awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host``awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner``CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail且 service `inactive``MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service若 registration 缺失readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail不要手工反覆補同一組 artifact。

View File

@@ -1,9 +1,9 @@
# AWOOOI 重開機恢復 SOP
> **版本**: v5.17
> **版本**: v5.18
> **最後更新**: 2026-07-03 (台北時間)
> **更新者**: Codex
> **觸發事件**: 2026-07-03 15:57 Windows99 VMware autostart task exists/enabled 但 `0x41303` 顯示未觸發,五台 VM guest 仍 powered off
> **觸發事件**: 2026-07-03 17:18 Windows99 VMX source repair package CLI 冷啟動修正111 VMX missing 轉入 check-mode repair lane
---
@@ -21,6 +21,12 @@
## 架構概覽與依賴圖
### 2026-07-03 17:18 P0-4 VMX Repair CLI 覆蓋
Gitea CD `#4617` 已完成 source `40713b2f8` 投產deploy marker `b85d574`production deploy readback matched=truejob succeeded。production reboot SLO scorecard 已讀回 Windows99 default target `111 / 188 / 120 / 121 / 112`,其中 `missing_vmx_aliases=["111"]``powered_off_aliases=["112","120","121","188","111"]`,舊 `110` 不得再當 Windows99 guest alias。
`build-windows99-vmx-source-repair-package.py` 是 no-secret / no-DB / check-mode 工具,必須能在工作站缺 `DATABASE_URL` 時輸出 `windows99-vmx-source-repair-111` package這只表示 repair check-mode package ready不表示 111 VMX 已恢復。via-110 locator 若回 `blocked_ssh_publickey_auth_missing``ssh_auth_probe_exit_status=124`,固定視為 management-channel blocker。下一步是取得 111 既有 VMX source / verified backup / 授權 console candidate identity建立 scoped relink dry-run、rollback 與 post-verifier不得猜 generic Ubuntu candidate、不得直接 relink、不得 restore、不得 power on VM、不得讀 Windows 密碼。
### 2026-07-03 15:57 P0-4 Windows99 Task Run-State 覆蓋
Windows99 no-secret via-110 collector 已讀回 `AWOOOI-Start-VMware-VMs` scheduled task`present=1``enabled=1``state=Ready``trigger_count=1``action_count=1`VMware service 與 Windows Update no-auto-reboot policy 也為 ready但新增 task detail 顯示 `last_run_time=1999-11-30T00:00:00``last_task_result=267011`,等價於 `0x41303`,表示 task 尚未跑過 / 未在本輪 boot 觸發。2026-07-03 16:33 進一步確認舊 verifier default 把 `110` 誤列為 Windows99 guest最新 source-of-truth 改為 `111 / 188 / 120 / 121 / 112`,以對齊本輪 host probe `111 unreachable` 與 99 VMware guest 需求。因此目前 P0-4 真 blocker 是 `windows99_vmware_guest_power_not_ready``windows99_vmware_autostart_task_never_ran`,且 deployment 後必須用新版 collector 重新讀回 111。

View File

@@ -22,7 +22,7 @@
| P0-1 | BLOCKED_REBOOT_EVENT_AND_HOST_REACHABILITY | 全主機 reboot auto-detection / auto-trigger / 10 分鐘恢復 SLO | 2026-07-03 15:49 verify-only artifact `/tmp/awoooi-reboot-event-verify-20260703-154907``reboot_detected=false``all_required_hosts_observed=false``all_required_hosts_in_reboot_window=false``recovery_deadline_status=target_window_elapsed``unreachable_hosts=["111"]`、99 reachable 但 `uptime_seconds=unknown`、188 reachable 但 `systemd_state=degraded` / `startup_active=failed`。production SLO 仍為 `blocked_reboot_auto_recovery_slo_not_ready`,不能宣稱 10 分鐘自動恢復。 | 固定先 rerun host probe + reboot-event detector verify-only並收斂 111 reachability、99 uptime / VMware readback、188 degraded/startup failed不得 reboot、不得 VM power change、不得讀 Windows 密碼,不得用 ping-only 或舊 boot evidence 當完成證據。 |
| P0-2 | DEPLOYED_RUNTIME_EDGE_DRIFT_BLOCKED | Deploy / reboot 期間 public 502 維護頁與外部 fallback | 2026-07-03 15:35 source fix 已部署讀回production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 保留 `edge_public_maintenance_live_config_drift``runtime_metric_active_blocker_count=8`,且 route fallback ready 不再清掉 edge Nginx drift。110 SLO artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-151219` 與 textfile 證明 edge 未 ready`awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"} 0`、active blocker `edge_public_maintenance_live_config_drift`。 | 後續才進 privileged edge apply 或 L1 cloud/CDN fallback decision完成條件是 edge fallback live config / maintenance file / `nginx -t` / reload / public header verifier 或外部 L1 fallback 全綠。不得讀密碼、不得用 app / Docker / host restart 掩蓋 edge fallback drift。 |
| P0-3 | PARTIAL_GREEN_DEPLOYED_SOURCE | 所有產品 / 網站版本與資料最新性 | Gitea `main=bc8d3af3f chore(cd): deploy dacdd90 [skip ci]`CD `#4558` log 顯示 API/Web build/push wrapper 全部 `rc=0`、deploy marker `dacdd90`、production deploy readback matched。Production SLO route 已讀回最新 API 行為:`runtime_metric_runtime_readback_added_blockers=["windows99_vmware_autostart_config_not_ready","windows99_update_no_auto_reboot_policy_not_ready"]``windows99_update_no_auto_reboot_ready=false`。Stock freshness 仍為 `status=ok``latest_trading_date=2026-07-02`、blockers `[]`。 | 版本層已補到 AWOOOI production完成整體 P0 仍需收斂 Windows99 VMX / service / policy / power 與 host reboot-event blockers。後續每個 public product 仍要維持 source SHA、deploy marker、runtime endpoint、public route watch、freshness 四層一致。 |
| P0-4 | BLOCKED_WINDOWS99_TASK_NOT_TRIGGERED | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | 2026-07-03 16:33 production readback 已證明 source/API/textfile 都會輸出 `windows99_vmware_autostart_task_never_ran`,但同時發現舊 source-of-truth 把 Windows99 default VM target 寫成 `110 / 188 / 120 / 121 / 112`,與本輪事故需求及 host probe `111 unreachable` 不一致。最新 source 修正為 `111 / 188 / 120 / 121 / 112`110 回到 P0 host / control-plane truth不再當 99 VMware guest default。 | 部署修正後重新跑 no-secret via-110 collector + SLO scorecard確認 `required_vm_aliases``powered_off_aliases`、check-mode package、validator 與 API readback 都以 111 為 Windows99 VM target。仍不得從 scorecard 直接 power on VM、重啟 Windows host / VMware service、改 scheduled task / registry 或讀 Windows 密碼。 |
| P0-4 | BLOCKED_WINDOWS99_VMX_SOURCE_AND_TASK_NOT_TRIGGERED | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | 2026-07-03 17:18 CD `#4617` 已完成 source `40713b2f8` 投產deploy marker `b85d574`production deploy readback matched=truejob succeeded。production scorecard 已讀回 `required_vm_aliases=["111","188","120","121","112"]``missing_vmx_aliases=["111"]``powered_off_aliases=["112","120","121","188","111"]`,舊 `110` 不再出現在 Windows99 VM alias。`build-windows99-vmx-source-repair-package.py` 已修正可在無 `DATABASE_URL` 冷啟動環境產生 `windows99-vmx-source-repair-111` packagevia-110 locator 本輪仍回 `blocked_ssh_publickey_auth_missing` / `ssh_auth_probe_exit_status=124`。 | 下一步固定為取得 111 既有 VMX source / verified backup / 授權 console candidate identity產生 scoped relink dry-run + rollback + post-verifier再 rerun no-secret collector + SLO scorecard。仍不得從 scorecard 直接 power on VM、重啟 Windows host / VMware service、改 scheduled task / registry、猜 generic Ubuntu candidate、restore 到 production 或讀 Windows 密碼。 |
| P0-5 | RUNTIME_READY_BACKUP_RECEIPT_GAP | Gitea / 主機 / DB / 網站 / 服務 / 套件 / 工具 / log 備份監控告警 | Gitea repo bundle readback readyexpected `12`、rows `12`、missing `0`、failed `0`、sample restore dry-run okbackup core green。2026-07-03 source / runtime 已部署 `awoooi_backup_alert_receipt_*` 指標與 Prometheus rules110 exporter 讀回 88 個 stage requirement、188 讀回 12 個 stage requirement`BackupAlertReceiptMetricMissing*` inactive`BackupAlertReceiptStageMissing` 已修成每 `host / receipt_channel` 聚合 pending110 一條、188 一條。 | 補 `/backup/alert-receipts/*.last_success` 脫敏 marker下一層仍要補 Gitea full dump、DB/settings/issues/packages/LFS、所有工具與 log 全量備份監控。 |
| P0-6 | RUNTIME_READY_ALERT_RECEIPT_GAP | 主機關機 / 重啟 / SLO miss / backup failure Telegram 告警 | Reboot per-blocker alert 與 backup receipt alert rules 已 deploy/readbackbackup receipt 缺段不再產生 100 條 stage 噪音,現在聚合成 110 / 188 兩條 host-level pending。scorecard 目前有 10 個 reboot active blockers其中 Windows99 config / policy blocker 已可供 per-blocker alert routing 使用;尚未完成 shutdown / reboot / backup alert 的 production 脫敏 delivery receipt 全矩陣。 | 補 alert receipt readbackhost down、host up、SLO miss、Windows99 blocker、backup stale/failed、deploy 502、freshness stale完成條件是每類告警都有 sent / received / dedup / escalation evidence。 |
| P0-7 | SOURCE_READY_SLA_AUTOMATION | 固定排查順序、ETA / wait reason、自動化判斷與修復 | Scorecard 已固定 `current_phase``eta_or_wait_reason``primary_blocker`、fixed triage order 與 action matrix2026-07-03 新增 `windows99_vmware_autostart_task_never_ran`,其 next action 固定為 `run_windows99_vmware_autostart_check_mode_package_for_powered_off_aliases_then_rerun_no_secret_collector_no_vm_power_change`,不再退回泛用「再收一次 verify」。 | 把每個 blocker 的 next_safe_action、post_verifier、forbidden_actions 接到自動 work item / Telegram / scorecard完成條件是重啟後自動判斷、主動告警、主動 rerun verifier不再人工臨場猜流程。 |

View File

@@ -10,6 +10,7 @@ from __future__ import annotations
import argparse
import json
import os
import sys
import urllib.request
from pathlib import Path
@@ -20,6 +21,11 @@ _API_ROOT = _REPO_ROOT / "apps" / "api"
if str(_API_ROOT) not in sys.path:
sys.path.insert(0, str(_API_ROOT))
# This CLI is a no-DB, check-mode reboot recovery tool. The API package import
# still validates settings, so provide a harmless placeholder when running
# outside the service container.
os.environ.setdefault("DATABASE_URL", "postgresql+asyncpg://test:test@localhost/test")
from src.services.reboot_auto_recovery_slo_scorecard import ( # noqa: E402
_build_windows99_vmx_source_repair_package,
_taipei_now_iso,

View File

@@ -2,6 +2,7 @@ from __future__ import annotations
import importlib.util
import json
import os
import subprocess
from pathlib import Path
@@ -172,6 +173,7 @@ def test_cli_writes_package_file(tmp_path: Path) -> None:
capture_output=True,
text=True,
check=False,
env={key: value for key, value in os.environ.items() if key != "DATABASE_URL"},
)
assert result.returncode == 0