diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 9a5f932ca..f6192627b 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,19 @@ +## 2026-07-03 — 14:26 Windows99 runtime metric overlay production readback 收斂 + +**完成內容**: +- 已把 commit `33c391808 fix(reboot): prefer windows99 runtime metrics` 推上 Gitea main,Gitea CD `#4589` 成功,deploy marker `9f49651 chore(cd): deploy 33c3918 [skip ci]` 已 fast-forward 回本機。 +- Public Gitea queue readback 顯示 `current_main_cd_run_status=Success`、`latest_visible_cd_production_deploy_readback_matched_from_log=true`、`latest_visible_cd_job_succeeded_from_log=true`、`latest_visible_cd_run_level_lag_after_deploy_success=false`、`no_matching_online_runner_visible=false`,且沒有 host pressure refusal。 +- 110 latest SLO run `reboot-auto-recovery-slo-20260703-142122` 已完成,service `Result=success`、`ExecMainStatus=0`;textfile metric 已輸出 `awoooi_reboot_auto_recovery_slo_blocker_count=7`、`awoooi_windows99_vmware_autostart_config_ready=1`、`awoooi_windows99_vmware_windows_update_no_auto_reboot_ready=1`,missing VMX alias 為空,powered-off aliases 是 `110,112,120,121,188`。 +- Production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已對齊 runtime truth:`active_blocker_count=7`、`runtime_metric_active_blocker_count=7`、`runtime_metric_runtime_readback_added_blocker_count=0`、`readiness_percent=67`、`primary_blocker=reboot_event_required_host_unreachable`,不再出現 Windows99 config / Windows Update policy 假 blocker。 +- `FULL-STACK-COLD-START-SOP` 升到 v1.121,`REBOOT-RECOVERY-SOP` 升到 v5.13,固定記錄部署後 production readback 與剩餘 7 個真 blocker。 + +**仍維持**: +- P0-006 尚未完成:不能宣稱全主機重啟後 10 分鐘內全部恢復;剩餘真 blocker 是 fresh all-host reboot event / 10-minute window / host reachability / `reboot_event_required_host_unreachable` / Windows99 guest power readback。 +- 未讀 secret / token / `.env` / raw sessions / SQLite / auth;未使用 GitHub / gh;未 workflow_dispatch;未啟動或關閉 VM;未重啟 host / service;未 Docker / Nginx / K3s / DB / firewall restart;未 DROP / TRUNCATE / restore / prune / delete / force push。 + +**下一步**: +- 繼續 P0-006:以同一套 scorecard/action matrix 補 fresh all-host reboot event / host reachability / Windows99 powered-off guest check-mode package;不得再回頭追已清除的 VMX missing、config 或 Windows Update policy 假 blocker。 + ## 2026-07-03 — 14:10 Windows99 runtime metric overlay 防 stale snapshot 假 blocker **完成內容**: diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 85b99e2cd..a07f1ec75 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,6 +1,6 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.120 +> Version: v1.121 > Last updated: 2026-07-03 Asia/Taipei > Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號,但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal;此納入不代表授權任何破壞性 runtime apply。 @@ -72,6 +72,8 @@ v1.119 Windows99 no-secret verifier via-host / validator rule:本工作站直 v1.120 Windows99 runtime metric overlay precedence rule:v1.110 的 nested blocker promotion 只適用於 runtime metric 缺少 Windows99 config / policy 欄位,或 metric 明確回 not ready 的情況。`reboot-auto-recovery-slo-exporter.sh` 必須從同一份 latest `scorecard.json` 輸出 `awoooi_windows99_vmware_autostart_config_ready`、`awoooi_windows99_vmware_windows_update_no_auto_reboot_ready`、`awoooi_windows99_vmware_missing_vmx_alias` 與 `awoooi_windows99_vmware_powered_off_alias`。production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 讀到 runtime metric `config_ready=1`、`windows_update_no_auto_reboot_ready=1`、`missing_vmx_aliases=[]` 時,不得再用 committed snapshot 的舊 `service_blockers` / `policy_blockers` 反加 `windows99_vmware_autostart_config_not_ready` 或 `windows99_update_no_auto_reboot_policy_not_ready`。若 `runtime_metric_active_blocker_count=7` 但 API `active_blocker_count` 被 stale snapshot 放大,優先修 API overlay / exporter metric,不得把假 blocker 轉成 Windows service restart、registry apply、VM power change、host reboot、Docker / Nginx / K3s / DB / firewall restart 或 secret 讀取。 +v1.121 deployed Windows99 runtime metric overlay readback rule:2026-07-03 14:26 production readback 已證明 source `33c391808` 的 API overlay 與 110 textfile metric 對齊。Gitea CD `#4589` 為 `Success`,deploy marker `9f49651`,production deploy readback matched;110 最新 SLO run `20260703-142122` 回 `Result=success`,textfile metric 為 `awoooi_reboot_auto_recovery_slo_blocker_count=7`、`awoooi_windows99_vmware_autostart_config_ready=1`、`awoooi_windows99_vmware_windows_update_no_auto_reboot_ready=1`、missing VMX alias 空、powered-off aliases `110 / 112 / 120 / 121 / 188`。production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 必須讀回 `active_blocker_count=7`、`runtime_metric_active_blocker_count=7`、`runtime_metric_runtime_readback_added_blocker_count=0`、`readiness_percent=67`、`primary_blocker=reboot_event_required_host_unreachable`;若再次出現 Windows99 config / Windows Update policy blocker,視為 API / Prometheus / textfile regression 先修 readback,不得跳去 power on VM、改 Windows policy、重啟 host 或重啟服務。這不代表 10 分鐘自動恢復 SLO 完成;剩餘真 blocker 仍是 fresh all-host reboot event / 10-minute window / host reachability / Windows99 guest power readback。 + 2026-07-02 110 control-path / Harbor recovery receipt rule:若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure,但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata,不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier,並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示:public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`;active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure,以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker,non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker;non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。 2026-07-02 110 controlled CD lane fail-closed enforcer staging rule:110 runner 壓力事故後,legacy / generic runner 仍必須 fail-closed;但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail,且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit,並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service;若 registration 缺失,readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail,不要手工反覆補同一組 artifact。 diff --git a/docs/runbooks/REBOOT-RECOVERY-SOP.md b/docs/runbooks/REBOOT-RECOVERY-SOP.md index 3a6cf3b17..6208f8c0d 100644 --- a/docs/runbooks/REBOOT-RECOVERY-SOP.md +++ b/docs/runbooks/REBOOT-RECOVERY-SOP.md @@ -1,9 +1,9 @@ # AWOOOI 重開機恢復 SOP -> **版本**: v5.12 +> **版本**: v5.13 > **最後更新**: 2026-07-03 (台北時間) > **更新者**: Codex -> **觸發事件**: 2026-07-03 14:10 Windows99 runtime metric overlay:110 SLO textfile 已證明 VMX missing 清除、config / Windows Update ready;API 不得用 stale snapshot 反加假 blocker +> **觸發事件**: 2026-07-03 14:26 Windows99 runtime metric overlay 已部署並 production readback 對齊:API / runtime metric 均為 7 個真 blocker,Windows99 config / Windows Update 假 blocker 已清除 --- @@ -21,6 +21,24 @@ ## 架構概覽與依賴圖 +### 2026-07-03 14:26 P0 現況覆蓋 + +Gitea CD `#4589` 已完成 source `33c391808` 部署,deploy marker `9f49651 chore(cd): deploy 33c3918 [skip ci]` 已 fast-forward 回 `gitea-ssh/main`;public queue readback 顯示 `current_main_cd_run_status=Success`、`current_main_cd_deploy_evidence_succeeded_from_log=true`、`latest_visible_cd_production_deploy_readback_matched_from_log=true`、`latest_visible_cd_job_succeeded_from_log=true`、`no_matching_online_runner_visible=false`,且沒有 host pressure refusal。110 `awoooi-reboot-auto-recovery-slo.service` 最新 run `20260703-142122` 已 `Result=success`、`ExecMainStatus=0`,textfile metric 顯示 `awoooi_reboot_auto_recovery_slo_blocker_count=7`、`awoooi_windows99_vmware_autostart_config_ready=1`、`awoooi_windows99_vmware_windows_update_no_auto_reboot_ready=1`、`awoooi_windows99_vmware_missing_vmx_alias` 無 alias gauge,powered-off aliases 為 `110 / 112 / 120 / 121 / 188`。 + +Production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 已與 runtime metric 對齊:`status=blocked_reboot_auto_recovery_slo_not_ready`、`active_blocker_count=7`、`runtime_metric_active_blocker_count=7`、`runtime_metric_runtime_readback_added_blocker_count=0`、`readiness_percent=67`、`primary_blocker=reboot_event_required_host_unreachable`、`can_claim_all_services_recovered_within_target=false`。`windows99_vmware_autostart_config_not_ready` 與 `windows99_update_no_auto_reboot_policy_not_ready` 已不得再列為目前 blocker;若未來再次出現,優先判定為 metric/API overlay regression,不得轉成 Windows policy apply、service restart、VM power change 或 host reboot。 + +目前仍不得宣稱「全主機重啟後 10 分鐘內全部恢復」。剩餘 7 個 active blockers 固定為: + +- `all_required_hosts_not_in_10_minute_reboot_window` +- `fresh_all_host_reboot_event_missing` +- `host_boot_observation_older_than_target_window` +- `host_unreachable_after_reboot` +- `host_uptime_unknown` +- `reboot_event_required_host_unreachable` +- `windows99_vmware_guest_power_not_ready` + +固定下一步:維持 110 timer / textfile / production scorecard readback,優先補 fresh all-host reboot event / 10-minute window evidence / host reachability,以及 Windows99 powered-off guest 的 check-mode package 與 post-verifier;禁止讀 Windows 密碼、啟動或關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。 + ### 2026-07-03 14:10 P0 現況覆蓋 目前仍不得宣稱「全主機重啟後 10 分鐘內全部恢復」。Gitea CD `#4587` 已完成 source `30c89a0` 部署並推 deploy marker `4da6dda`;110 上 `awoooi-reboot-auto-recovery-slo.timer` 已 `enabled/active`,最新 textfile runtime metric 為 `active_blocker_count=7`,primary blocker `reboot_event_required_host_unreachable`,`can_claim_all_services_recovered_within_target=false`。7 個 runtime active blockers 固定為: