fix(cd): correct recovery profile path and stock guardrail [metadata-only]
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 40s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 40s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped
This commit is contained in:
@@ -51968,6 +51968,28 @@ production browser smoke:
|
||||
- rebase 到 Gitea main 最新 deploy marker,commit / push 本次 source 修法與 receipt。
|
||||
- 讀回 Gitea CD / Prometheus;下一條主線是把 `RebootAutoRecoverySLOMissed` 的 6 個 blocker 轉成可自動判斷與可回滾 remediation,不消音告警。
|
||||
|
||||
## 2026-07-02 — 00:24 P0 110 sustained moderate CPU alert 實際降壓
|
||||
|
||||
**完成內容**:
|
||||
- Prometheus 主動觸發 `Host110SustainedModeratePressure`,證明 110 CPU 壓力已被監控抓到;這次沒有等使用者人工發現後才開始排查。
|
||||
- 讀回 `awoooi_host_load5_per_core{host="110"}=0.786667`,orphan browser group `0`,active Gitea Actions process count / CPU 均為 `0`,swap ratio 約 `0.01273`。
|
||||
- top container 顯示 `stockplatform-v2-postgres-1` 曾達約 `292.96% CPU`,且 inspect 為 `nanocpus=0 memory=0 memswap=0 restart=no`;這是 110 host pressure 的實際風險,不是噪音。
|
||||
- 已 live 套用:`docker update --cpus=1.5 --memory=4g --memory-swap=6g --restart unless-stopped stockplatform-v2-postgres-1`。此操作只改 Docker cgroup / restart policy metadata,沒有重啟 DB container,沒有 DB write。
|
||||
- post-apply inspect:`running=true health=healthy nanocpus=1500000000 memory=4294967296 memswap=6442450944 restart=unless-stopped`。
|
||||
- direct Docker stats after apply:`stockplatform-v2-postgres-1 cpu=149.38% mem=506.2MiB / 4GiB`。
|
||||
- Prometheus 收斂後:`stockplatform-v2-postgres-1=1.6508 cores`、`gitea=1.8097 cores`、`load5/core=0.673333`。
|
||||
- StockPlatform public route / health / data freshness 未受損:`/` 200、`/healthz` 200、`/api/healthz` 200、`/api/v1/system/freshness` 200 且 `status=ok`、`latest_trading_date=2026-07-01`。
|
||||
- `Host110SustainedModeratePressure` 已從 firing alerts 清除;目前 firing alerts 只剩本次 reboot lane 的 `HostRebootEventDetected` 與 `RebootAutoRecoverySLOMissed`。
|
||||
- `ops/reboot-recovery/full-stack-cold-start-baseline.yml` 已補 `stockplatform_v2_postgres` resource guardrail,避免下次重啟回退成無限額。
|
||||
|
||||
**仍維持**:
|
||||
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth。
|
||||
- 沒有使用 GitHub / gh / GitHub API / GitHub Actions。
|
||||
- 沒有重啟主機,沒有 Docker / Nginx / K3s / DB / firewall restart,沒有 workflow_dispatch,沒有 DROP / TRUNCATE / restore / prune / query kill / manual DB write。
|
||||
|
||||
**下一步**:
|
||||
- 以 `[skip ci]` source commit 推回 Gitea main,避免只剩 live drift;然後回到 Gitea queue / 110 control-path / reboot SLO blocker 主線。
|
||||
|
||||
## 2026-07-02 — P0 AI loop stale control-path blocker production readback 修正
|
||||
|
||||
**完成內容**:
|
||||
|
||||
@@ -81,6 +81,33 @@
|
||||
"awoooi_reboot_auto_recovery_slo_blocker_count": 6,
|
||||
"awoooi_reboot_event_target_seconds_remaining": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"host": "110",
|
||||
"target": "stockplatform-v2-postgres-1",
|
||||
"action": "docker update --cpus=1.5 --memory=4g --memory-swap=6g --restart unless-stopped stockplatform-v2-postgres-1",
|
||||
"blast_radius": "container resource guardrail only; no container restart and no database write",
|
||||
"readback": {
|
||||
"running": true,
|
||||
"health": "healthy",
|
||||
"nanocpus": 1500000000,
|
||||
"memory_bytes": 4294967296,
|
||||
"memswap_bytes": 6442450944,
|
||||
"restart_policy": "unless-stopped",
|
||||
"direct_docker_stats_after": "cpu=149.38% mem=506.2MiB / 4GiB",
|
||||
"prometheus_after_convergence": {
|
||||
"stockplatform_v2_postgres_cpu_cores": 1.6508,
|
||||
"gitea_cpu_cores": 1.8097,
|
||||
"host_110_load5_per_core": 0.673333
|
||||
},
|
||||
"public_route_health": {
|
||||
"root": 200,
|
||||
"healthz": 200,
|
||||
"api_healthz": 200,
|
||||
"freshness_status": "ok",
|
||||
"latest_trading_date": "2026-07-01"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"prometheus_readback": {
|
||||
@@ -101,7 +128,8 @@
|
||||
"DockerContainerMemoryLimitPressure",
|
||||
"HostOutOfDiskSpace",
|
||||
"HostDiskUsageHigh",
|
||||
"RebootEventDetectorMissing"
|
||||
"RebootEventDetectorMissing",
|
||||
"Host110SustainedModeratePressure"
|
||||
],
|
||||
"filesystem_used_percent": {
|
||||
"110": 79.68293081715491,
|
||||
|
||||
@@ -2472,3 +2472,30 @@ python3 scripts/ops/docker-disk-pressure-retention-cleanup.py --host-label 188 -
|
||||
- 可以宣稱:這次回讀的 runtime full-stack 已 GREEN,110 / 188 disk pressure、momo-db missing resource limit、Sentry relay memory pressure、RebootEventDetectorMissing 已解除。
|
||||
- 不可宣稱:這次重啟有達成 10 分鐘 SLO。Prometheus 仍正確保留 `RebootAutoRecoverySLOMissed`,代表本次事件已 missed,必須做為後續自動化改善 evidence,而不是消音。
|
||||
- 不可用 stale API snapshot 覆蓋 live truth;`https://aiops.wooo.work/api/v1/agents/reboot-auto-recovery-slo-scorecard` 若 `generated_at` 舊於最新 textfile / Prometheus readback,只能列為 stale artifact。
|
||||
|
||||
### 16.4 2026-07-02 110 StockPlatform DB pressure guardrail
|
||||
|
||||
2026-07-02 00:20 後,`Host110SustainedModeratePressure` 正確觸發。這不是缺監控;Prometheus 已主動抓到 110 壓力。即時 evidence 顯示 orphan browser group `0`、active Gitea Actions process `0`、swap ratio 約 `0.0127`,壓力來源是 `gitea` 接近既有 2 CPU cap,以及 `stockplatform-v2-postgres-1` 沒有 Docker resource guardrail、曾瞬間到約 `292.96% CPU`。
|
||||
|
||||
已完成 live 修復:
|
||||
|
||||
```bash
|
||||
docker update --cpus=1.5 --memory=4g --memory-swap=6g --restart unless-stopped stockplatform-v2-postgres-1
|
||||
```
|
||||
|
||||
讀回證據:
|
||||
|
||||
| Check | Evidence |
|
||||
|-------|----------|
|
||||
| Docker inspect | `running=true health=healthy nanocpus=1500000000 memory=4294967296 memswap=6442450944 restart=unless-stopped` |
|
||||
| Direct Docker stats | `stockplatform-v2-postgres-1 cpu=149.38% mem=506.2MiB / 4GiB` after apply |
|
||||
| Prometheus convergence | `stockplatform-v2-postgres-1=1.6508 cores`、`gitea=1.8097 cores`、`load5/core=0.673333` |
|
||||
| Product route | `https://stock.wooo.work/`、`/healthz`、`/api/healthz` all `200` |
|
||||
| Product data freshness | `/api/v1/system/freshness` `status=ok`、`latest_trading_date=2026-07-01` |
|
||||
| Alert readback | `Host110SustainedModeratePressure` cleared;firing alerts only `HostRebootEventDetected` and `RebootAutoRecoverySLOMissed` |
|
||||
|
||||
宣告限制:
|
||||
|
||||
- 可宣稱:110 sustained moderate CPU alert 已主動觸發、已定位、已用 Docker guardrail 降壓,StockPlatform route / health / freshness 未受損。
|
||||
- 不可宣稱:110 永久不會再高負載;Gitea queue / hook backlog 仍需在後續 queue / control-path lane 持續觀察。
|
||||
- 不得用 DB restart、container restart、manual DB write、query kill 或 fake freshness 來處理 StockPlatform pressure。
|
||||
|
||||
Reference in New Issue
Block a user