docs(reboot): anchor current slo readback [metadata-only]
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 57s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped

This commit is contained in:
Your Name
2026-07-03 10:54:43 +08:00
parent c7bb7d4ffd
commit f69e056bab
4 changed files with 41 additions and 11 deletions

View File

@@ -1,3 +1,18 @@
## 2026-07-03 — 10:53 Reboot SLO current readback anchor 校正
**完成內容**
- 重新讀回 production `/api/v1/agents/reboot-auto-recovery-slo-scorecard`,確認 `https://aiops.wooo.work``https://awoooi.wooo.work` 兩個 public route 一致:`active_blocker_count=10``readiness_percent=60``status=blocked_reboot_auto_recovery_slo_not_ready``can_claim_all_services_recovered_within_target=false`
- 目前 active blockers 固定包含 reboot event / host reachability / uptime / Windows99 VMX / guest power / VMware config / Windows Update policy`windows99_missing_vmx_aliases=["111"]``windows99_powered_off_aliases=["111","112","120","121","188"]`
- 重新產生 live no-secret package`/tmp/awoooi-p0-006-windows99-vmx-package-codex-current/package.json`,狀態為 `check_mode_package_ready_windows99_vmx_source_repair_required``apply_allowed_by_this_package=false`111 expected VMX path 為 `D:\Documents\Virtual Machines\192.168.0.111_Ubuntu_64-bit\192.168.0.111_Ubuntu_64-bit.vmx`
- 修正 `docs/runbooks/REBOOT-RECOVERY-SOP.md``docs/runbooks/FULL-STACK-COLD-START-SOP.md` 與統帥插入需求台帳:不得再把 08:23 的 `8 / 67%` 當目前狀態;目前下一步固定為 `restore_windows99_missing_vmx_source_for_aliases_then_rerun_no_secret_collector_and_scorecard_no_vm_power_change`
**已跑驗證**
- Public scorecard readback`aiops.wooo.work` / `awoooi.wooo.work` 兩路一致。
- `python3.11 scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py --timeout-seconds 25 --output /tmp/awoooi-p0-006-windows99-vmx-package-codex-current/package.json`:成功產生 no-secret check-mode package。
**仍維持**
- 未讀 secret / token / `.env` / raw sessions / SQLite / auth未使用 GitHub / gh未 workflow_dispatch未啟動或關閉 VM未重啟 host / service未 Docker / Nginx / K3s / DB / firewall restart未 DROP / TRUNCATE / restore / prune / delete / force push。
## 2026-07-03 — 10:48 Windows99 VMX source repair check-mode package
**完成內容**

View File

@@ -1,6 +1,6 @@
# AWOOOI 全棧冷啟動與主機重啟 SOP
> Version: v1.111
> Version: v1.112
> Last updated: 2026-07-03 Asia/Taipei
> Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號,但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal此納入不代表授權任何破壞性 runtime apply。
@@ -54,6 +54,8 @@ v1.110 Windows99 nested runtime blocker promotion ruleproduction `/api/v1/age
v1.111 Gitea CD Docker step bounded-timeout rulepublic queue 顯示 `cd.yaml` Running 不得視為 production 已更新或中性等待;必須同時讀 `latest_visible_cd_deploy_marker_*``latest_visible_cd_production_deploy_readback_matched_from_log``latest_visible_cd_failure_classifier``latest_visible_cd_inflight_classifier``latest_visible_cd_docker_step_timeout*``Build and Push API` / `Build and Push Web``docker build``docker push` 必須由 `run_docker_step` 包住,預設 API build `1200s`、Web build `1500s`、push `600s`timeout 時 fail-closed 輸出 `BLOCKER cd_docker_step_timeout label=... timeout_seconds=...``read-public-gitea-actions-queue.py --json` 必須把此 blocker 上卷為 `blocked_cd_docker_step_timeout` 與固定 safe-next-action下一步只能檢查 runner Docker build/push path、registry route 與 queue readback不得用 workflow_dispatch、重啟 Docker / host / Nginx / K3s / DB、讀 secrets 或恢復 generic runner 掩蓋卡點。
v1.112 current production SLO readback anchor2026-07-03 10:50 production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 的目前事實是 `status=blocked_reboot_auto_recovery_slo_not_ready``active_blocker_count=10``readiness_percent=60``can_claim_all_services_recovered_within_target=false``primary_blocker=reboot_event_required_host_unreachable`。active blockers 固定為 `all_required_hosts_not_in_10_minute_reboot_window``fresh_all_host_reboot_event_missing``host_boot_observation_older_than_target_window``host_unreachable_after_reboot``host_uptime_unknown``reboot_event_required_host_unreachable``windows99_vmware_guest_power_not_ready``windows99_vmware_vmx_missing``windows99_vmware_autostart_config_not_ready``windows99_update_no_auto_reboot_policy_not_ready``runtime_metric_runtime_readback_added_blockers` 固定含 Windows99 config / policy 兩項,`windows99_update_no_auto_reboot_ready=false`。下一步固定為 `restore_windows99_missing_vmx_source_for_aliases_then_rerun_no_secret_collector_and_scorecard_no_vm_power_change`。不得再用 08:23 的 `8 / 67%` 當目前狀態;它只能保留為歷史 readback。此錨點仍只授權 source / verifier / check-mode package不授權 VM power change、Windows service restart、registry apply、host reboot、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete 或 secret 讀取。
2026-07-02 110 control-path / Harbor recovery receipt rule若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/``401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh``recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335``Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` markernon110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blockernon110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。
2026-07-02 110 controlled CD lane fail-closed enforcer staging rule110 runner 壓力事故後legacy / generic runner 仍必須 fail-closed`awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host``awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner``CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail且 service `inactive``MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service若 registration 缺失readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail不要手工反覆補同一組 artifact。

View File

@@ -1,9 +1,9 @@
# AWOOOI 重開機恢復 SOP
> **版本**: v5.4
> **版本**: v5.5
> **最後更新**: 2026-07-03 (台北時間)
> **更新者**: Codex
> **觸發事件**: 2026-07-03 08:23 reboot verify-only readback99 no-secret management channel blocked、111 unreachable、188 startup failed/degraded、backup receipt runtime gap
> **觸發事件**: 2026-07-03 10:50 production SLO scorecard readbackWindows99 VMX / guest power / service config / Windows Update policy blocker 已上卷10 分鐘 SLO 仍 blocked
---
@@ -21,9 +21,22 @@
## 架構概覽與依賴圖
### 2026-07-03 08:23 P0 現況覆蓋
### 2026-07-03 10:50 P0 現況覆蓋
目前不得宣稱「全主機重啟後 10 分鐘內全部恢復」。production SLO scorecard 仍為 `blocked_reboot_auto_recovery_slo_not_ready``active_blocker_count=8``readiness_percent=67`、primary blocker `reboot_event_required_host_unreachable`。最新 no-write artifact 是 `/tmp/awoooi-reboot-verify-only-20260703-082310`
目前不得宣稱「全主機重啟後 10 分鐘內全部恢復」。production SLO scorecard 仍為 `blocked_reboot_auto_recovery_slo_not_ready``active_blocker_count=10``readiness_percent=60`、primary blocker `reboot_event_required_host_unreachable``can_claim_all_services_recovered_within_target=false`。10 個 active blockers 固定為
- `all_required_hosts_not_in_10_minute_reboot_window`
- `fresh_all_host_reboot_event_missing`
- `host_boot_observation_older_than_target_window`
- `host_unreachable_after_reboot`
- `host_uptime_unknown`
- `reboot_event_required_host_unreachable`
- `windows99_vmware_guest_power_not_ready`
- `windows99_vmware_vmx_missing`
- `windows99_vmware_autostart_config_not_ready`
- `windows99_update_no_auto_reboot_policy_not_ready`
最新 no-write artifact 仍以 `/tmp/awoooi-reboot-verify-only-20260703-082310` 與後續 Windows99 management / collector readback 為來源:
- 99 可 ping / RDP / Hyper-V VMConnect但 uptime unknownSSH BatchMode 與 WinRM 仍不可用。
- 111 仍 unreachable。
@@ -31,7 +44,7 @@
- 188 可達,但 `systemd_state=degraded``awoooi-startup.service failed`
- `reboot_detected=false``fresh_boot_hosts=[]`,因此 10 分鐘 SLO 尚未證明。
固定下一步:恢復 99 no-secret management channel 或收集 console Verify stdout `windows99-vmware-autostart.ps1 -Mode Verify` 產生 normalized artifact同時補 111 reachability再重跑 host probe、reboot-event detector、SLO scorecard。禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。
固定下一步:`restore_windows99_missing_vmx_source_for_aliases_then_rerun_no_secret_collector_and_scorecard_no_vm_power_change`。先恢復 111 VMX source 與 99 no-secret management / console Verify stdout使 `windows99-vmware-autostart.ps1 -Mode Verify` 產生 normalized artifact同時補 111 reachability再重跑 host probe、reboot-event detector、SLO scorecard。禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。
### 五主機全貌

View File

@@ -57,14 +57,14 @@
| 順序 | ID | 優先序 | 使用者插入要求 | 正規化工作項 | 目前狀態 | 下一個可驗證動作 |
| --- | --- | --- | --- | --- | --- | --- |
| 1 | CIR-P0-RBT-001 | P0 | 「主機重啟後 10 分鐘內全部恢復,且要自動判斷所有主機被重啟」 | 建立 99/110/111/112/120/121/188 reboot event detector + 10 分鐘 SLO scorecard + fixed triage order | 2026-07-03 08:59 production scorecardreadiness `67%`、active blockers `8``can_claim_all_services_recovered_within_target=false`artifact `/tmp/awoooi-reboot-continue-20260703-085205`111 不可達、99 uptime unknown、188 startup failed/degraded09:22 已把 unreliable console artifact 投影成 machine-readable blocker | 優先收斂 99 local console Verify output / no-secret management channel 與 111 reachability不可宣稱 10 分鐘 SLA 已證明,不得用 RDP clipboard 片段當完成證據 |
| 1 | CIR-P0-RBT-001 | P0 | 「主機重啟後 10 分鐘內全部恢復,且要自動判斷所有主機被重啟」 | 建立 99/110/111/112/120/121/188 reboot event detector + 10 分鐘 SLO scorecard + fixed triage order | 2026-07-03 10:50 production scorecardreadiness `60%`、active blockers `10``can_claim_all_services_recovered_within_target=false`blockers 已含 reboot event / host unreachable / uptime unknown / Windows99 VMX / guest power / VMware config / Windows Update policyartifact `/tmp/awoooi-reboot-continue-20260703-085205`111 不可達、99 uptime unknown、188 startup failed/degraded09:22 已把 unreliable console artifact 投影成 machine-readable blocker | 優先收斂 111 VMX source、99 local console Verify output / no-secret management channel 與 111 reachability不可宣稱 10 分鐘 SLA 已證明,不得用 RDP clipboard 片段當完成證據 |
| 2 | CIR-P0-RBT-002 | P0 | 「沒有偵測到主機重啟」 | 修正 host reboot/shutdown/up detectionboot_id / uptime / node exporter / Windows exporter / VMware VM power state 都要進同一事件 | Scorecard 已接 collection packet + management probe08:23 host probe 有 7 列但 99 只有 ping/RDP reachable、uptime unknown111 unreachable112/120/121/188 uptime 都已超過 10 分鐘窗口 | 讓 99 verifier / Windows exporter 或等效 no-secret readback 進入 host boot event並補 111 reachability 證據 |
| 3 | CIR-P0-RBT-003 | P0 | 「192.168.0.99 VMWare 要自動啟動,裡面 111/188/120/121/112 也自動啟動」 | Windows 99 VMware host autostart + guest VM autostart contractVM host 111/188/120/121/112 開機順序與 readback | Source verifier / parser / API readback / collection packet 已完成10:15 source/API patch 已把 nested `service_blockers=["VMAuthdService","VMnetDHCP"]` 與 VMX / power evidence 提升到 active blocker matrix不再只顯示 guest power not ready09:53 management probe 讀回 99 host reachable、RDP / Hyper-V VMConnect reachable但 SSH BatchMode `permission_denied`、WinRM timeout`console_artifact_status=blocked_clipboard_unreliable` | 先修 111 VMX source 與 VMware service / autostart task check-mode package重跑 no-secret collector 與 scorecard不得從 scorecard 直接啟動 / 關閉 VM、重啟 Windows service、讀密碼或 host reboot |
| 4 | CIR-P0-RBT-004 | P0 | 「192.168.0.99 不可因 Windows Update 無預警重開」 | Windows Update reboot policyactive hours / no auto-restart / maintenance window / update notification audit | 10:15 source/API patch若 nested verifier 回 `policy_blockers=["windows_update_policy_readback_missing"]`API 必須 fail-closed 重新提升 `windows99_update_no_auto_reboot_policy_not_ready`;不得只信舊 Prometheus `windows99_update_no_auto_reboot_ready=true` | 補 no-secret policy readback / drift alert若 policy 不綠,另走 controlled apply package + rollback + post-verifier禁止要求或記錄 Windows 密碼,禁止由 scorecard 直接 apply registry |
| 5 | CIR-P0-RBT-005 | P0 | 「網站重啟後 502 嚴重影響體驗,要維護頁,外部雲端或專業做法」 | Public maintenance fallbackNginx / edge / external static maintenance page / status page / fail-open UX避免 502 直出 | Source + runtime verifier 已實作Gitea CD `#4459` Success、deploy marker `8d7a6faaf`production API scorecard 已讀回 `public_maintenance_fallback.ready=true`、raw 5xx=`0`、unreachable without L1=`0`P0 blockers `11`、readiness `47`;仍不可宣稱整體 10 分鐘 SLO 完成 | 若未來 public route 出現 raw 5xx先走 L0 Nginx intercept / header verifier若 edge unreachable才產生 L1 external static origin / CDN decision record + rollback |
| 6 | CIR-P0-RBT-006 | P0 | 「所有主機關機立刻 Telegram 告警,重啟後也要告警,其他告警一併完整思考」 | Down / shutdown suspected / reboot detected / reboot recovered / SLO missed / backup failed / freshness stale / CPU pressure / Gitea queue 告警矩陣 | HostDown / HostRebootEventDetected / RebootAutoRecoverySLOMissed 已存在per-blocker reboot alerts 與 backup receipt rules 已 deploy/readback。Backup receipt 缺段已從 100 條 stage 噪音收斂為 110 / 188 兩條 host-level pending仍需完整 shutdown/up E2E receipt | 補 Prometheus / Alertmanager active/resolved 與 outbound receiptbackup alert 先補 `/backup/alert-receipts/*.last_success` 脫敏 marker不送測試 secret、不重啟主機 |
| 7 | CIR-P0-RBT-007 | P0 | 「所有備份包含主機、DB、網站、服務、套件、工具、日誌都沒有監控告警」 | Backup observability coveragebackup job inventory、last success、freshness、offsite、restore drill、Telegram/AwoooP receipt | 已有 backup health exporter / alert rules / Gitea bundle restore dry-run2026-07-03 runtime 讀回 110 有 88 個 receipt stage requirement、188 有 12 個,`BackupAlertReceiptMetricMissing*` inactive`BackupAlertReceiptStageMissing` 聚合 pending 110 / 188 各一條 | 補 `/backup/alert-receipts/*.last_success`;再補 Gitea full dump / DB / settings / issues / packages / LFS 與所有工具/log 全量備份監控 |
| 8 | CIR-P0-RBT-008 | P0 | 「每次重啟排查都不一樣,也不知道多久恢復,不符合 SLA」 | 固定化 reboot runbookfixed triage order、ETA、active blocker、remaining seconds、owner lane、next command | Production scorecard readback 已固定 `status=blocked_reboot_auto_recovery_slo_not_ready`、readiness `67%`、active blockers `8`、primary `reboot_event_required_host_unreachable`09:22 source/API contract 已把 unreliable console artifact 接到 `active_blocker_action_matrix`owner lane 與 next safe action | 優先收斂 99 no-secret Verify / 111 reachability / 188 startup failed/degraded不得用不同排查路徑繞過 scorecard |
| 8 | CIR-P0-RBT-008 | P0 | 「每次重啟排查都不一樣,也不知道多久恢復,不符合 SLA」 | 固定化 reboot runbookfixed triage order、ETA、active blocker、remaining seconds、owner lane、next command | Production scorecard readback 已固定 `status=blocked_reboot_auto_recovery_slo_not_ready`、readiness `60%`、active blockers `10`、primary `reboot_event_required_host_unreachable`10:50 next action 固定為 `restore_windows99_missing_vmx_source_for_aliases_then_rerun_no_secret_collector_and_scorecard_no_vm_power_change` | 優先收斂 111 VMX source、99 no-secret Verify / 111 reachability / 188 startup failed/degraded不得用不同排查路徑繞過 scorecard |
| 9 | CIR-P0-RBT-009 | P0 | 「所有產品、網站都要是最新版本;版本和數據是否最新要驗證」 | Product freshness/version matrixsource commit、deploy marker、runtime image、public health、data freshness、latest source availability | AWOOOI Gitea `main=bc8d3af3f chore(cd): deploy dacdd90 [skip ci]`CD `#4558` log 顯示 API/Web build/push wrapper 全部 `rc=0`、deploy marker `dacdd90`、production deploy readback matched。Production SLO route 已讀回最新 API 行為:`active_blocker_count=10``runtime_metric_runtime_readback_added_blockers=["windows99_vmware_autostart_config_not_ready","windows99_update_no_auto_reboot_policy_not_ready"]``windows99_update_no_auto_reboot_ready=false`。StockPlatform public freshness / ingestion 仍為 `ok`latest trading date `2026-07-02`。 | 版本層已補到 AWOOOI production下一步仍要建立全產品 readback 表,且收斂 Windows99 VMX / service / policy / power 與 host reboot-event blockers |
| 10 | CIR-P0-GIT-001 | P0 | 「Gitea 儲存庫都不見了Gitea 沒完整備份嗎?」 | Gitea repository identity + backup proof + restore drill不能只看 UI visible要比對 SSH heads、repo path、bundle backup、restore sample | 2026-07-02 production `/api/v1/agents/gitea-repo-bundle-backup-readback` 已 ready9 expected repos present/ok、missing=0、failed=0、checksum_missing=0、bundle_fresh=true、all_expected_ok=true、sample_restore_dry_run_ok=truerepo bundle / restore dry-run 層已關閉,不是 repo missing。 | 維持每日 bundle backup + restore dry-run monitoring另補 Gitea full dump / DB / settings / issues / packages / LFS 備份 readback。禁止刪 repo / 改 visibility / 讀 token / restore 到 production |
| 11 | CIR-P0-CPU-001 | P0 | 「110 / 188 CPU 負載持續過高,為什麼沒監控告警、沒主動修復」 | Sustained CPU pressure automationAlertmanager → controller → evidence → service playbook → verifier → KM writeback | 110 已有 `Host110SustainedModeratePressure`、Gitea playbook、Stock/Postgres evidence188 仍需同級 controller/alerts readback | 下一步接 `postgres_hot_query_or_backup_export_playbook`;並補 188 equivalent readback不以單次下降結案 |
@@ -134,10 +134,10 @@
| KM / PlayBook / RAG / MCP 整合 | 已被列為 P1不再遺漏 | 建立 work item schema 與 trust writeback 欄位 |
| OpenClaw / Gather-style 持續動畫工作室 | route 已存在,已列為 P1 工作項 | 補 production desktop/mobile smoke、AwoooP 導流與截圖證據 |
| AI 專業 UI / 非文字牆 cockpit | 已列為 P2 UX 驗收 | 將長文字區塊收斂成 first-viewport cockpit、cards、flow rows 與 expandable details |
| 10 分鐘 reboot auto-recovery SLA | 2026-07-03 08:23 production scorecard 固定 `8` blockers / readiness `67%`,但 `can_claim_all_services_recovered_within_target=false`artifact `/tmp/awoooi-reboot-verify-only-20260703-082310` 保留 host probe / reboot detector / scorecard | 收斂 99 no-secret Verify、111 reachability、188 startup failed/degraded再 rerun host probe + reboot-event detector |
| 10 分鐘 reboot auto-recovery SLA | 2026-07-03 10:50 production scorecard 固定 `10` blockers / readiness `60%`,但 `can_claim_all_services_recovered_within_target=false`artifact `/tmp/awoooi-reboot-verify-only-20260703-082310` 保留 host probe / reboot detector / scorecardproduction readback 已把 Windows99 config / policy blockers 上卷 | 收斂 111 VMX source、99 no-secret Verify、111 reachability、188 startup failed/degraded再 rerun host probe + reboot-event detector |
| AWOOOI latest source production deploy readback | Gitea `main=bc8d3af3f`deploy marker `dacdd90`public CD `#4558` log job succeeded / production deploy readback matchedproduction scorecard 已讀回 latest API 行為並把 Windows99 config / policy blockers 上卷 | 下一步轉回 Windows99 VMX / service / policy / power 與 host reboot-event blockers不得因 deploy/readback 成功宣稱 10 分鐘 SLO 完成 |
| 99 Windows / VMware autostart | Source verifier / parser / API readback / collection packet 已完成live 99 Verify output 尚未收集08:23 management probe 顯示 RDP / Hyper-V VMConnect reachable但 SSH BatchMode / WinRM blockedcollector `blocked_ssh_publickey_auth_missing` | 收集 99 no-secret Verify output確認 VM 111/188/120/121/112 running、scheduled task / services / Windows Update policy 全綠 |
| Reboot SLO blocker 收斂 | production scorecard 已固定 `8` blockers / readiness `67%`source / runtime 已補 per-blocker metric/alert剩餘主 blocker 是 host boot detection + Windows99 VMware VMX / guest power | 依具名 blocker 收斂 99/111/188不得用 route green 或 RDP 可見宣稱 SLA 完成 |
| Reboot SLO blocker 收斂 | production scorecard 已固定 `10` blockers / readiness `60%`source / runtime 已補 per-blocker metric/alert剩餘主 blocker 是 host boot detection + Windows99 VMware VMX / guest power / autostart config / Windows Update policy | 依具名 blocker 收斂 99/111/188不得用 route green 或 RDP 可見宣稱 SLA 完成 |
| 全備份監控告警 coverage | exporter/rule 已有 host/DB/site/service/package/tool/log coverage 與 backup alert receipt requirementruntime rules 已部署,缺段 alert 聚合成 110 / 188 host-level pendingproduction receipt markers / full Gitea dump / DB/settings/issues/packages/LFS 尚未全綠 | 補 `/backup/alert-receipts/*.last_success`、Gitea full dump 與 restore drill verifier |
| Stock/Postgres hot pressure | 110 live 已導向 Stock/Postgres playbook尚未完成 hot query / backup export playbook closure | 下一步執行 read-only Stock/Postgres evidence 與 source freshness / query attribution |
@@ -154,7 +154,7 @@
## 下一輪固定執行順序
1. 先確認 Gitea `main` / CD / production readback 是否仍與最新 truth 一致。
2. 接續目前 active P0-006用 08:23 artifact 的固定順序收斂 99 no-secret Verify、111 reachability、188 startup failed/degraded再 rerun host probe / reboot-event detector / scorecard。
2. 接續目前 active reboot P0用 10:50 production scorecard 的固定順序收斂 111 VMX source、99 no-secret Verify、111 reachability、188 startup failed/degraded再 rerun host probe / reboot-event detector / scorecard。
3. 補全 reboot auto-recovery P099/110/111/112/120/121/188 reboot detection、10 分鐘 SLO scorecard、Telegram down/up/recovered/SLO missed / active-blocker receipts。
4. 補 99 Windows / VMware autostart P099 host 自動啟動 VMwareVM 111/188/120/121/112 自動啟動;同時補 Windows Update no-auto-restart verifier。
5. 補 111 reachability / host boot readback`host_unreachable_after_reboot``reboot_event_required_host_unreachable` 有單一固定處理路徑。