fix(reboot): expose fixed recovery sop progress
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 51s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled
AI 技術雷達監控 / ai-technology-watch (push) Successful in 38s
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 51s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled
AI 技術雷達監控 / ai-technology-watch (push) Successful in 38s
This commit is contained in:
@@ -64,7 +64,7 @@
|
||||
| 5 | CIR-P0-RBT-005 | P0 | 「網站重啟後 502 嚴重影響體驗,要維護頁,外部雲端或專業做法」 | Public maintenance fallback:Nginx / edge / external static maintenance page / status page / fail-open UX,避免 502 直出 | 尚未完整落地;目前是需求缺口 | 產生 `public_maintenance_fallback` decision record:DNS/edge/外部雲端/本地 Nginx fallback 風險比較,先做不切流量的 check-mode |
|
||||
| 6 | CIR-P0-RBT-006 | P0 | 「所有主機關機立刻 Telegram 告警,重啟後也要告警,其他告警一併完整思考」 | Down / shutdown suspected / reboot detected / reboot recovered / SLO missed / backup failed / freshness stale / CPU pressure / Gitea queue 告警矩陣 | 部分已有 Alertmanager rule 與 Telegram receipt 補強;仍缺完整 shutdown/up E2E receipt | 建立 Telegram alert matrix + receipt verifier,逐項讀回 Alertmanager active/resolved 與 outbound receipt,不送測試 secret |
|
||||
| 7 | CIR-P0-RBT-007 | P0 | 「所有備份包含主機、DB、網站、服務、套件、工具、日誌都沒有監控告警」 | Backup observability coverage:backup job inventory、last success、freshness、offsite、restore drill、Telegram receipt | 部分已有 backup health exporter / alert rules;全域 coverage 與 restore drill 未全綠 | 建立 backup coverage matrix:host / DB / website / service config / package list / tool scripts / logs,每列有 metric、alert、last_success、restore_verifier |
|
||||
| 8 | CIR-P0-RBT-008 | P0 | 「每次重啟排查都不一樣,也不知道多久恢復,不符合 SLA」 | 固定化 reboot runbook:fixed triage order、ETA、active blocker、remaining seconds、owner lane、next command | 部分已有 scorecard / SOP;仍需所有回報統一格式 | 將 SLO scorecard 強制輸出 `current_phase`、`eta_or_wait_reason`、`active_blockers`、`next_safe_action` |
|
||||
| 8 | CIR-P0-RBT-008 | P0 | 「每次重啟排查都不一樣,也不知道多久恢復,不符合 SLA」 | 固定化 reboot runbook:fixed triage order、ETA、active blocker、remaining seconds、owner lane、next command | 已補 scorecard source / snapshot / API 固定欄位:`current_phase`、`eta_or_wait_reason`、`primary_blocker`、`next_safe_action`;仍需 production deploy readback 與下一次 fresh reboot/drill 證明 | 推送後讀回 `/api/v1/agents/reboot-auto-recovery-slo-scorecard`,確認固定欄位在 production 可見;再接 99/VMware 與 fresh reboot/drill |
|
||||
| 9 | CIR-P0-RBT-009 | P0 | 「所有產品、網站都要是最新版本;版本和數據是否最新要驗證」 | Product freshness/version matrix:source commit、deploy marker、runtime image、public health、data freshness、latest source availability | AWOOOI / StockPlatform 部分已在做;全產品未統一 | 建立全產品 readback 表:product、canonical repo、main SHA、deploy marker、public URL、data freshness、blocked reason |
|
||||
| 10 | CIR-P0-GIT-001 | P0 | 「Gitea 儲存庫都不見了?Gitea 沒完整備份嗎?」 | Gitea repository identity + backup proof + restore drill:不能只看 UI visible,要比對 SSH heads、repo path、bundle backup、restore sample | 已有 9 expected repos OK / backup health missing=0 的 handoff;仍需 restore drill 證明 | 補 Gitea repo bundle backup readback + sample restore dry-run verifier;禁止刪 repo / 改 visibility |
|
||||
| 11 | CIR-P0-CPU-001 | P0 | 「110 / 188 CPU 負載持續過高,為什麼沒監控告警、沒主動修復」 | Sustained CPU pressure automation:Alertmanager → controller → evidence → service playbook → verifier → KM writeback | 110 已有 `Host110SustainedModeratePressure`、Gitea playbook、Stock/Postgres evidence;188 仍需同級 controller/alerts readback | 下一步接 `postgres_hot_query_or_backup_export_playbook`;並補 188 equivalent readback,不以單次下降結案 |
|
||||
@@ -103,7 +103,7 @@
|
||||
| KM / PlayBook / RAG / MCP 整合 | 已被列為 P1,不再遺漏 | 建立 work item schema 與 trust writeback 欄位 |
|
||||
| OpenClaw / Gather-style 持續動畫工作室 | route 已存在,已列為 P1 工作項 | 補 production desktop/mobile smoke、AwoooP 導流與截圖證據 |
|
||||
| AI 專業 UI / 非文字牆 cockpit | 已列為 P2 UX 驗收 | 將長文字區塊收斂成 first-viewport cockpit、cards、flow rows 與 expandable details |
|
||||
| 10 分鐘 reboot auto-recovery SLA | SLO exporter / alerts 部分存在,但缺 fresh all-host reboot/drill proof | 補最新 scorecard readback,缺事件則明確標等待下一次 reboot 或 approved drill |
|
||||
| 10 分鐘 reboot auto-recovery SLA | scorecard source / snapshot / API 已補固定 phase / ETA / blocker / next action 欄位;仍缺 fresh all-host reboot/drill proof | 補 production API readback,缺事件則明確標等待下一次 reboot 或 approved drill |
|
||||
| 99 Windows / VMware autostart | 尚未完成 99 host + VM 111/188/120/121/112 autostart verifier | 建立 non-secret VMware/Windows verifier,不讀密碼 |
|
||||
| 502 maintenance fallback | 尚未完成外部維護頁 / edge fallback 決策與實作 | 先做 no-write decision record + smoke verifier |
|
||||
| 全備份監控告警 coverage | 部分 exporter/rule 已存在,但 host/DB/site/service/package/tool/log coverage 未全列 | 建立 backup coverage matrix 與 restore drill verifier |
|
||||
|
||||
Reference in New Issue
Block a user