fix(ops): close post-reboot recovery guardrails

2026-07-02 00:18:17 +08:00
parent 6b8edd8ffe
commit a15ab298ff
6 changed files with 654 additions and 2 deletions
--- a/docs/LOGBOOK.md
+++ b/docs/LOGBOOK.md
@@ -51934,6 +51934,40 @@ production browser smoke:
 - 沒有使用 GitHub / gh / GitHub API / GitHub Actions。
 - 沒有重啟主機，沒有 Docker / Nginx / K3s / DB restart，沒有 workflow_dispatch，沒有 DROP / TRUNCATE / restore / prune。

+## 2026-07-02 — 00:17 P0 2026-07-01 post-reboot runtime recovery GREEN 與 SLO miss 留證
+
+**完成內容**：
+- 188 `momo-db` 補上 Docker resource guardrail：`--cpus=2 --memory=4g --memory-swap=6g`；live inspect 讀回 `nanocpus=2000000000 memory=4294967296 memswap=6442450944 restart=unless-stopped`，`https://mo.wooo.work/health` 回 `status=healthy / database=postgresql / version=V10.725`。
+- 110 `sentry-self-hosted-relay-1` memory-limit pressure 修復：live inspect 讀回 `health=healthy memory=3221225472 memswap=4294967296`，`https://sentry.wooo.work/_health/` 回 `ok`。
+- 新增 `scripts/ops/docker-disk-pressure-retention-cleanup.py` 與測試，提供 dry-run first、bounded、非 destructive 的 Docker disk-pressure cleanup；明確禁止 `docker system prune`，不碰 containers / volumes / DB / backups。
+- 110 disk pressure 從約 `85%` 降到 Prometheus `79.6829%`；188 disk pressure 從約 `85%` 降到 Prometheus `79.2250%`。
+- 重新安裝 110 reboot auto-recovery SLO textfile，scope 從舊 `110_120_121_188` 修正為 `99_110_111_112_120_121_188`；live textfile 讀回 `awoooi_reboot_event_detected=1`，`RebootEventDetectorMissing` 已清除。
+- Prometheus firing alerts 從 7 個降到 2 個，只剩 `HostRebootEventDetected` 與 `RebootAutoRecoverySLOMissed`；disk/resource/missing-detector 類告警已清除。
+- 全主機 cold-start scorecard 已重新跑：`PASS=96 WARN=0 BLOCKED=0`，result `GREEN`。
+- `ops/reboot-recovery/full-stack-cold-start-baseline.yml` 已補 source baseline：110 Sentry relay `3g/4g swap`，188 `momo-db 2CPU/4G/6G swap/unless-stopped`，避免 live 修復下次回退。
+- 新增 machine-readable receipt：`docs/operations/post-reboot-runtime-recovery-readback-2026-07-01.snapshot.json`。
+- `docs/runbooks/FULL-STACK-COLD-START-SOP.md` 新增 §16.3，沉澱本次 resource / disk / detector / full-stack GREEN 與 SLO miss 宣告界線。
+
+**本地驗證結果**：
+- `python3.11 -m pytest scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py -q`：`6 passed`。
+- `python3.11 -m pytest scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py scripts/ops/tests/test_host_pressure_alert_contract.py -q`：`9 passed`。
+- `python3.11 -m py_compile scripts/ops/docker-disk-pressure-retention-cleanup.py scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py`：通過。
+- live full-stack：`SSH_CONNECT_TIMEOUT=8 bash scripts/reboot-recovery/full-stack-cold-start-check.sh --monitor-read-only --no-color` → `PASS=96 WARN=0 BLOCKED=0`。
+
+**live truth**：
+- 可以宣稱目前 runtime full-stack recovered / GREEN。
+- 不可宣稱本次重啟達成 10 分鐘 SLO；Prometheus 保留 `RebootAutoRecoverySLOMissed` 是正確 incident evidence，不應消音。
+- `https://aiops.wooo.work/api/v1/agents/reboot-auto-recovery-slo-scorecard` 仍可能是 stale snapshot；目前以 live textfile / Prometheus / cold-start scorecard 為 runtime truth。
+
+**仍維持**：
+- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth；沒有讀 `.runner` 內容。
+- 沒有使用 GitHub / gh / GitHub API / GitHub Actions。
+- 沒有重啟主機，沒有 Docker / Nginx / K3s / DB / firewall restart，沒有 workflow_dispatch，沒有 DROP / TRUNCATE / restore / prune / remote delete。
+
+**下一步**：
+- rebase 到 Gitea main 最新 deploy marker，commit / push 本次 source 修法與 receipt。
+- 讀回 Gitea CD / Prometheus；下一條主線是把 `RebootAutoRecoverySLOMissed` 的 6 個 blocker 轉成可自動判斷與可回滾 remediation，不消音告警。
+
 ## 2026-07-01 — 23:28 P0 110 sustained CPU pressure alert / controlled quota / alert-chain readback

 **完成內容**：
--- a/docs/operations/post-reboot-runtime-recovery-readback-2026-07-01.snapshot.json
+++ b/docs/operations/post-reboot-runtime-recovery-readback-2026-07-01.snapshot.json
@@ -0,0 +1,151 @@
+{
+  "schema_version": "awoooi_post_reboot_runtime_recovery_readback_v1",
+  "generated_at": "2026-07-02T00:17:36+08:00",
+  "scope": "99_110_111_112_120_121_188",
+  "purpose": "Record live post-reboot recovery evidence after 2026-07-01 all-host reboot and prevent source drift.",
+  "boundaries": {
+    "read_secret_values": false,
+    "read_env_files": false,
+    "read_raw_sessions": false,
+    "read_sqlite": false,
+    "used_github": false,
+    "rebooted_hosts": false,
+    "restarted_docker_nginx_k3s_db_firewall": false,
+    "used_docker_system_prune": false,
+    "touched_containers": false,
+    "touched_volumes": false,
+    "touched_databases": false,
+    "touched_backups": false
+  },
+  "live_actions": [
+    {
+      "host": "188",
+      "target": "momo-db",
+      "action": "docker update --cpus=2 --memory=4g --memory-swap=6g momo-db",
+      "blast_radius": "container resource guardrail only; no container restart",
+      "readback": {
+        "running": true,
+        "nanocpus": 2000000000,
+        "memory_bytes": 4294967296,
+        "memswap_bytes": 6442450944,
+        "restart_policy": "unless-stopped",
+        "public_health": "https://mo.wooo.work/health returned status=healthy database=postgresql version=V10.725"
+      }
+    },
+    {
+      "host": "110",
+      "target": "sentry-self-hosted-relay-1",
+      "action": "docker update --memory=3g --memory-swap=4g sentry-self-hosted-relay-1",
+      "blast_radius": "container resource guardrail only; no container restart",
+      "readback": {
+        "running": true,
+        "health": "healthy",
+        "memory_bytes": 3221225472,
+        "memswap_bytes": 4294967296,
+        "public_health": "https://sentry.wooo.work/_health/ returned ok"
+      }
+    },
+    {
+      "host": "110",
+      "target": "root filesystem Docker pressure",
+      "action": "bounded dangling image and BuildKit cache cleanup with scripts/ops/docker-disk-pressure-retention-cleanup.py",
+      "blast_radius": "unreferenced dangling images and builder cache only",
+      "readback": {
+        "filesystem_used_percent_after_prometheus": 79.68293081715491,
+        "direct_df_after": "/dev/mapper/ubuntu--vg-ubuntu--lv ext4 983G Used 743G Avail 200G Use% 79% /",
+        "docker_images_after": "92.91GB total, 29.81GB reclaimable",
+        "docker_build_cache_after": "43.57GB total, 12.59GB reclaimable"
+      }
+    },
+    {
+      "host": "188",
+      "target": "root filesystem Docker pressure",
+      "action": "bounded BuildKit cache cleanup with scripts/ops/docker-disk-pressure-retention-cleanup.py",
+      "blast_radius": "builder cache only in final zero-age pass",
+      "readback": {
+        "filesystem_used_percent_after_prometheus": 79.22498388062007,
+        "direct_df_after": "/dev/mapper/ubuntu--vg-ubuntu--lv ext4 982G Used 737G Avail 204G Use% 79% /",
+        "docker_images_after": "42.93GB total, 14.96GB reclaimable",
+        "docker_build_cache_after": "33.31GB total, 42.82MB reclaimable"
+      }
+    },
+    {
+      "host": "110",
+      "target": "reboot auto recovery SLO textfile",
+      "action": "bash scripts/reboot-recovery/install-reboot-auto-recovery-slo-110.sh",
+      "blast_radius": "script/timer/textfile install and verifier run only",
+      "readback": {
+        "scope": "99_110_111_112_120_121_188",
+        "awoooi_reboot_event_detected": 1,
+        "awoooi_reboot_auto_recovery_slo_ready": 0,
+        "awoooi_reboot_auto_recovery_slo_blocker_count": 6,
+        "awoooi_reboot_event_target_seconds_remaining": 0
+      }
+    }
+  ],
+  "prometheus_readback": {
+    "alerts_after_recovery": [
+      {
+        "alertname": "HostRebootEventDetected",
+        "severity": "warning",
+        "host": "110"
+      },
+      {
+        "alertname": "RebootAutoRecoverySLOMissed",
+        "severity": "critical",
+        "scope": "99_110_111_112_120_121_188"
+      }
+    ],
+    "cleared_alert_classes": [
+      "DockerContainerMissingResourceLimit",
+      "DockerContainerMemoryLimitPressure",
+      "HostOutOfDiskSpace",
+      "HostDiskUsageHigh",
+      "RebootEventDetectorMissing"
+    ],
+    "filesystem_used_percent": {
+      "110": 79.68293081715491,
+      "188": 79.22498388062007
+    }
+  },
+  "cold_start_scorecard": {
+    "command": "SSH_CONNECT_TIMEOUT=8 bash scripts/reboot-recovery/full-stack-cold-start-check.sh --monitor-read-only --no-color",
+    "pass": 96,
+    "warn": 0,
+    "blocked": 0,
+    "result": "GREEN",
+    "runtime_truth": "Full stack is ready for controlled runner/CD release.",
+    "declaration_limit": "This proves current runtime recovery, not that the 10-minute reboot SLO was met."
+  },
+  "source_updates": [
+    "scripts/ops/docker-disk-pressure-retention-cleanup.py",
+    "scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py",
+    "ops/reboot-recovery/full-stack-cold-start-baseline.yml",
+    "docs/runbooks/FULL-STACK-COLD-START-SOP.md",
+    "docs/LOGBOOK.md"
+  ],
+  "local_verification": [
+    {
+      "command": "python3.11 -m pytest scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py -q",
+      "result": "6 passed"
+    },
+    {
+      "command": "python3.11 -m pytest scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py scripts/ops/tests/test_host_pressure_alert_contract.py -q",
+      "result": "9 passed"
+    },
+    {
+      "command": "python3.11 -m py_compile scripts/ops/docker-disk-pressure-retention-cleanup.py scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py",
+      "result": "passed"
+    }
+  ],
+  "remaining_truth": {
+    "runtime_full_stack_green": true,
+    "ten_minute_slo_met_for_this_reboot": false,
+    "why": "Reboot event was detected after the target window had elapsed, so Prometheus correctly keeps RebootAutoRecoverySLOMissed as incident evidence.",
+    "next_mainline": [
+      "Persist resource guardrails in host compose/source where owned by the matching repo.",
+      "Replace stale reboot SLO API snapshot with live textfile/Prometheus backed readback.",
+      "Convert the six reboot SLO blockers into automatic preflight/remediation steps without silencing the SLO miss."
+    ]
+  }
+}
--- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md
+++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md
@@ -2427,3 +2427,48 @@ DATABASE_URL=sqlite+aiosqlite:////tmp/awoooi-codex-api-test.db PYTHONPATH=apps/a
 bash -n scripts/ci/wait-host-web-build-pressure.sh scripts/ops/systemd-units-textfile-exporter.py
 python3.11 -m pytest scripts/ops/tests/test_systemd_units_textfile_exporter.py ops/runner/test_cd_controlled_runtime_profile.py -q
 ```
+
+### 16.3 2026-07-01 post-reboot resource / disk / detector closure
+
+2026-07-01 全主機重啟後，runtime 問題分成四條：188 `momo-db` 缺 Docker resource limit、110 Sentry relay memory-limit pressure、110 / 188 root filesystem 超過 disk-pressure 門檻、以及 reboot detector scope 仍停在舊 `110_120_121_188`。這些不是噪音；disk pressure 會阻塞 Docker / registry / build path，resource-limit pressure 會讓告警長期失真，detector scope 錯誤會讓「有沒有偵測到重啟」本身不可信。
+
+已完成 live 修復：
+
+| Layer | 修復 | 讀回證據 |
+|-------|------|----------|
+| 188 `momo-db` guardrail | `docker update --cpus=2 --memory=4g --memory-swap=6g momo-db` | `nanocpus=2000000000 memory=4294967296 memswap=6442450944 restart=unless-stopped` |
+| 110 Sentry relay guardrail | `docker update --memory=3g --memory-swap=4g sentry-self-hosted-relay-1` | `health=healthy memory=3221225472 memswap=4294967296`，`https://sentry.wooo.work/_health/` 回 `ok` |
+| 110 disk pressure | 只清 unreferenced dangling images 與 bounded builder cache；不使用 `docker system prune` | `/` 從約 `85%` 降到 Prometheus `79.6829%` |
+| 188 disk pressure | builder-cache-only follow-up；不碰 containers / volumes / DB / backups | `/` 從約 `85%` 降到 Prometheus `79.2250%` |
+| reboot detector | 安裝新版 auto-recovery SLO textfile scope | `awoooi_reboot_event_detected{scope="99_110_111_112_120_121_188"} 1` |
+| full-stack scorecard | 重新跑全主機 cold-start | `PASS=96 WARN=0 BLOCKED=0`，result `GREEN` |
+
+新的 source-of-truth：
+
+- `scripts/ops/docker-disk-pressure-retention-cleanup.py`
+- `scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py`
+- `ops/reboot-recovery/full-stack-cold-start-baseline.yml`
+- `docs/operations/post-reboot-runtime-recovery-readback-2026-07-01.snapshot.json`
+
+Docker disk pressure cleanup 規則：
+
+- 預設 dry-run；只有明確 `--apply` 才執行。
+- 不使用 `docker system prune`。
+- 不刪除 containers、volumes、databases、backups、logs。
+- dangling image cleanup 只允許移除未被任何 container 參照、沒有 tag、超過 age gate 的 image，且保留最新 N 個。
+- BuildKit cache cleanup 必須明確加 `--include-builder-cache`，並設定 `--builder-keep-storage`。
+- `--min-age-hours=0` 只允許和 `--skip-dangling-images` 一起使用，避免無 age gate 刪 image。
+
+標準命令：
+
+```bash
+python3 scripts/ops/docker-disk-pressure-retention-cleanup.py --host-label 110 --include-builder-cache
+python3 scripts/ops/docker-disk-pressure-retention-cleanup.py --host-label 110 --apply --min-age-hours 24 --keep-dangling-newest 20 --include-builder-cache --builder-keep-storage 30GB
+python3 scripts/ops/docker-disk-pressure-retention-cleanup.py --host-label 188 --skip-dangling-images --apply --min-age-hours 0 --include-builder-cache --builder-keep-storage 1GB
+```
+
+宣告限制：
+
+- 可以宣稱：這次回讀的 runtime full-stack 已 GREEN，110 / 188 disk pressure、momo-db missing resource limit、Sentry relay memory pressure、RebootEventDetectorMissing 已解除。
+- 不可宣稱：這次重啟有達成 10 分鐘 SLO。Prometheus 仍正確保留 `RebootAutoRecoverySLOMissed`，代表本次事件已 missed，必須做為後續自動化改善 evidence，而不是消音。
+- 不可用 stale API snapshot 覆蓋 live truth；`https://aiops.wooo.work/api/v1/agents/reboot-auto-recovery-slo-scorecard` 若 `generated_at` 舊於最新 textfile / Prometheus readback，只能列為 stale artifact。