fix(reboot): apply l0 maintenance fallback source
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 1s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Failing after 2m25s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped

This commit is contained in:
Your Name
2026-07-02 21:39:30 +08:00
parent c68b746861
commit 5b0c77495b
7 changed files with 70 additions and 12 deletions

View File

@@ -1,3 +1,15 @@
## 2026-07-02 — 21:49 P0-006 L0 public maintenance fallback source apply
**完成內容**
- 接續 Gitea CD `#4451` success / deploy marker `c68b74686` / production source `5d5bc86fed` readbackpublic post-rollout probe 讀回 6 條 public routes 都是 `200`raw 5xx without fallback=`0`
- 同一輪 CD build log 仍觀察到 `public_health_argocd_wait_http=502`,所以不能只停在 runtime probe將 AWOOOI edge Nginx source 從 legacy `/50x.html` 改成 L0 maintenance fallback。
- `ops/nginx/awoooi.wooo.work.conf``k8s/nginx/awoooi-prod.conf` 已加入 `proxy_intercept_errors on``error_page 502 503 504 /__awoooi-maintenance.html``/var/www/maintenance/maintenance.html``Retry-After``X-AWOOOI-Fallback=local-maintenance`
- `scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py` 新增 source contractAWOOOI Nginx source 不得退回 `/50x.html`,必須保留 L0 fallback header 與 maintenance file path。
- `PUBLIC-MAINTENANCE-FALLBACK-RUNBOOK.md` 與 commander priority ledger 已同步:目前是 source apply 完成runtime 仍需 edge staging vhost / `nginx -t` / header readback通過前不得宣稱 production fallback fully closed。
**仍維持**
- 未使用 GitHub / `gh` / GitHub API未讀 secret / token / `.env` / raw sessions / SQLite / auth未 workflow_dispatch未重啟主機 / VM / Docker / Nginx / K3s / DB / firewall未切 DNS / CDN / provider。
## 2026-07-02 — 21:38 P0-006 public maintenance fallback test fixture closure
**完成內容**

View File

@@ -17,7 +17,7 @@
## L0 受控套用
1.`ops/maintenance/maintenance.html` 部署到 edge host 的 `/var/www/maintenance/maintenance.html`
2. 在每個 public `server {}` 內 include `ops/maintenance/nginx-502-maintenance-snippet.conf` 的等價內容。
2. 在每個 public `server {}` 內 include `ops/maintenance/nginx-502-maintenance-snippet.conf` 的等價內容AWOOOI source conf `ops/nginx/awoooi.wooo.work.conf``k8s/nginx/awoooi-prod.conf` 已固定 `proxy_intercept_errors on``/__awoooi-maintenance.html``X-AWOOOI-Fallback: local-maintenance`
3. `nginx -t` 通過後才 reload Nginx。
4. 用壞 upstream staging vhost 或暫存 upstream 驗證 `X-AWOOOI-Fallback: local-maintenance`,不得用 production DB / app restart 當測試手段。
@@ -59,3 +59,16 @@ python3 scripts/reboot-recovery/reboot-auto-recovery-slo-scorecard.py \
- `public_route_raw_5xx_without_maintenance_fallback`L0 Nginx intercept 未生效,下一步只能做 staging vhost / `nginx -t` / header verifier不得用 production app restart 當測試。
- `public_route_unreachable_without_external_l1_fallback`edge 或外部路徑整段不可達,下一步是 L1 external static origin / CDN health check decision record 與 rollback不直接切 DNS。
- `public_maintenance_fallback_runtime_readback_missing`:沒有 runtime probe 證據,不得宣稱 public maintenance fallback 已完成。
## 2026-07-02 L0 source apply
已將 AWOOOI edge Nginx source 從 legacy `/50x.html` 改為 L0 maintenance fallback
- `ops/nginx/awoooi.wooo.work.conf`
- `k8s/nginx/awoooi-prod.conf`
- `ops/maintenance/nginx-502-maintenance-snippet.conf`
驗收仍必須分兩層:
- Source verifier`scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py` 必須確認上述 source 不可退回 `/50x.html`
- Runtime verifier在 edge host 用 staging vhost / `nginx -t` / header readback 證明 `X-AWOOOI-Fallback=local-maintenance`;通過前不得宣稱 production runtime 已接管 raw 502。

View File

@@ -61,11 +61,11 @@
| 2 | CIR-P0-RBT-002 | P0 | 「沒有偵測到主機重啟」 | 修正 host reboot/shutdown/up detectionboot_id / uptime / node exporter / Windows exporter / VMware VM power state 都要進同一事件 | Scorecard 已接 collection packet + management probe99 host reachable 但 uptime unknown111 unreachablestale hosts 仍存在 | 讓 99 verifier / Windows exporter 或等效 no-secret readback 進入 host boot event並補 111 reachability 證據 |
| 3 | CIR-P0-RBT-003 | P0 | 「192.168.0.99 VMWare 要自動啟動,裡面 111/188/120/121/112 也自動啟動」 | Windows 99 VMware host autostart + guest VM autostart contractVM host 111/188/120/121/112 開機順序與 readback | Source verifier / parser / API readback / collection packet 已完成management probe 讀回 `host_reachable=true`、RDP open、`2179` VMConnect / console channel open、SSH BatchMode `permission_denied`、WinRM timeoutsnapshot active blockers=`windows99_remote_execution_channel_unavailable``windows99_vmware_autostart_readback_missing` | 收集 local console Verify output 或恢復 no-secret management channel再確認 `VMRUN_PRESENT`、scheduled task、VMware services、VM power、VMX present 全綠 |
| 4 | CIR-P0-RBT-004 | P0 | 「192.168.0.99 不可因 Windows Update 無預警重開」 | Windows Update reboot policyactive hours / no auto-restart / maintenance window / update notification audit | Source verifier 已補 `WINDOWS_UPDATE_POLICY``WINDOWS_UPDATE_NO_AUTO_REBOOT_READY`collection packet 已列 forbidden actions99 management channel 尚不能收 policy readback | 取得 Verify output若 policy 不綠,再走 controlled apply禁止要求或記錄 Windows 密碼 |
| 5 | CIR-P0-RBT-005 | P0 | 「網站重啟後 502 嚴重影響體驗,要維護頁,外部雲端或專業做法」 | Public maintenance fallbackNginx / edge / external static maintenance page / status page / fail-open UX避免 502 直出 | Source + runtime verifier 已實作`public-maintenance-fallback-probe.py` 會抓 raw 5xx / L0 header / L1 unreachablescorecard 會 fail-closedL0 Nginx 實際套用與 L1 external/CDN cutover 尚需 runtime readback | 先讀回 production probe若出現 `public_route_raw_5xx_without_maintenance_fallback`,只做 staging vhost / `nginx -t` / header verifier若 edge unreachable產生 L1 external static origin / CDN decision record + rollback |
| 5 | CIR-P0-RBT-005 | P0 | 「網站重啟後 502 嚴重影響體驗,要維護頁,外部雲端或專業做法」 | Public maintenance fallbackNginx / edge / external static maintenance page / status page / fail-open UX避免 502 直出 | Source + runtime verifier 已實作AWOOOI Nginx source conf 已從 legacy `/50x.html` 改成 L0 `proxy_intercept_errors on` + `__awoooi-maintenance.html` + `X-AWOOOI-Fallback=local-maintenance`production post-rollout probe 目前 6 條 public routes 都是 `200`raw 5xx=`0`,但 CD rollout 期間仍觀察到 `public_health_argocd_wait_http=502`,所以 runtime 接管尚未驗收 | 下一步只做 edge staging vhost / `nginx -t` / header verifier通過後才 controlled reload若 edge unreachable產生 L1 external static origin / CDN decision record + rollback |
| 6 | CIR-P0-RBT-006 | P0 | 「所有主機關機立刻 Telegram 告警,重啟後也要告警,其他告警一併完整思考」 | Down / shutdown suspected / reboot detected / reboot recovered / SLO missed / backup failed / freshness stale / CPU pressure / Gitea queue 告警矩陣 | 部分已有 Alertmanager rule 與 Telegram receipt 補強;仍缺完整 shutdown/up E2E receipt | 建立 Telegram alert matrix + receipt verifier逐項讀回 Alertmanager active/resolved 與 outbound receipt不送測試 secret |
| 7 | CIR-P0-RBT-007 | P0 | 「所有備份包含主機、DB、網站、服務、套件、工具、日誌都沒有監控告警」 | Backup observability coveragebackup job inventory、last success、freshness、offsite、restore drill、Telegram receipt | 部分已有 backup health exporter / alert rules全域 coverage 與 restore drill 未全綠 | 建立 backup coverage matrixhost / DB / website / service config / package list / tool scripts / logs每列有 metric、alert、last_success、restore_verifier |
| 8 | CIR-P0-RBT-008 | P0 | 「每次重啟排查都不一樣,也不知道多久恢復,不符合 SLA」 | 固定化 reboot runbookfixed triage order、ETA、active blocker、remaining seconds、owner lane、next command | Scorecard/API contract 已固定 `next_safe_action=restore_windows99_no_secret_management_channel_or_collect_local_console_verify_readback_then_rerun_reboot_scorecard_no_reboot`;尚待 deploy readback | 推送後讀回 `/api/v1/agents/reboot-auto-recovery-slo-scorecard`,確認 production 顯示 99 management-channel blocker 與 readiness `43%` |
| 9 | CIR-P0-RBT-009 | P0 | 「所有產品、網站都要是最新版本;版本和數據是否最新要驗證」 | Product freshness/version matrixsource commit、deploy marker、runtime image、public health、data freshness、latest source availability | 本輪 StockPlatform public freshness / ingestion 讀回 `ok`latest trading date `2026-07-01`;全產品版本矩陣仍未統一 | 建立全產品 readback 表product、canonical repo、main SHA、deploy marker、public URL、data freshness、blocked reason |
| 8 | CIR-P0-RBT-008 | P0 | 「每次重啟排查都不一樣,也不知道多久恢復,不符合 SLA」 | 固定化 reboot runbookfixed triage order、ETA、active blocker、remaining seconds、owner lane、next command | Production scorecard readback 已固定 `status=blocked_reboot_auto_recovery_slo_not_ready`、readiness `43%`、active blockers `11``next_safe_action=restore_windows99_no_secret_management_channel_or_collect_local_console_verify_readback_then_rerun_reboot_scorecard_no_reboot` | 優先收斂 99 no-secret Verify / 111 reachability / backup_core / service_green / Wazuh degraded不得用不同排查路徑繞過 scorecard |
| 9 | CIR-P0-RBT-009 | P0 | 「所有產品、網站都要是最新版本;版本和數據是否最新要驗證」 | Product freshness/version matrixsource commit、deploy marker、runtime image、public health、data freshness、latest source availability | AWOOOI Gitea main 已到 deploy marker `c68b74686`production source readback `5d5bc86fed` verifiedStockPlatform public freshness / ingestion 讀回 `ok`latest trading date `2026-07-02`core price/chips/margin/AI recommendations 都是 `2026-07-02` | 建立全產品 readback 表product、canonical repo、main SHA、deploy marker、public URL、data freshness、blocked reason |
| 10 | CIR-P0-GIT-001 | P0 | 「Gitea 儲存庫都不見了Gitea 沒完整備份嗎?」 | Gitea repository identity + backup proof + restore drill不能只看 UI visible要比對 SSH heads、repo path、bundle backup、restore sample | 已有 9 expected repos OK / backup health missing=0 的 handoff仍需 restore drill 證明 | 補 Gitea repo bundle backup readback + sample restore dry-run verifier禁止刪 repo / 改 visibility |
| 11 | CIR-P0-CPU-001 | P0 | 「110 / 188 CPU 負載持續過高,為什麼沒監控告警、沒主動修復」 | Sustained CPU pressure automationAlertmanager → controller → evidence → service playbook → verifier → KM writeback | 110 已有 `Host110SustainedModeratePressure`、Gitea playbook、Stock/Postgres evidence188 仍需同級 controller/alerts readback | 下一步接 `postgres_hot_query_or_backup_export_playbook`;並補 188 equivalent readback不以單次下降結案 |
| 12 | CIR-P0-CPU-002 | P0 | 「噪音會影響真問題,要整合一起做」 | Alert noise / real issue correlationbackup aggregate noise、CPU pressure、Gitea queue、Stock freshness 要分清主因與次因 | 部分已在 SOP 註記;仍需統一 correlation scorecard | 建立 incident correlation readbackprimary_blocker、secondary_noise、ignored_noise_reason、evidence_ref |