docs(ops): record nginx exporter recovery [skip ci]
This commit is contained in:
Your Name
@@ -1,6 +1,6 @@
|
||||
# AWOOOI 全棧冷啟動與主機重啟 SOP
|
||||
|
||||
> Version: v1.33
|
||||
> Version: v1.34
|
||||
> Last updated: 2026-06-24 Asia/Taipei
|
||||
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
|
||||
|
||||
@@ -10,17 +10,18 @@
|
||||
|
||||
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
|
||||
|
||||
2026-06-24 18:32 notification-noise hardening supersedes the earlier 11:35 wording where it discusses heartbeat / MOMO alert behavior. The service and data readiness gates below remain unchanged until a fresh live cold-start scorecard says otherwise:
|
||||
2026-06-24 20:17 notification-noise hardening and 188 `nginx-exporter` recovery supersede the earlier 11:35 wording where it discusses heartbeat / MOMO alert behavior and monitoring coverage. The service and data readiness gates below are refreshed by the 20:17 live cold-start scorecard:
|
||||
|
||||
```text
|
||||
Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%.
|
||||
Live cold-start read-only check: PASS=86 WARN=0 BLOCKED=1, Result=BLOCKED.
|
||||
Service state: SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, ArgoCD awoooi-prod Synced/Healthy at revision 7db7800e399caed5487a705c81ec993dec76c70f, public routes/TLS green, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared.
|
||||
Runtime release state: API/Web/Worker are ready; image remains a84a5a0b because 7db7800e is docs-only and does not rebuild runtime images.
|
||||
MOMO state: mo.wooo.work health is healthy on version V10.639; current-month daily_sales_snapshot and realtime_sales_monthly match, but both stop at 2026-06-17. MOMO_DAILY_FRESHNESS is 7 days, which is a hard blocker because business data is not current.
|
||||
Runtime release state: API/Web/Worker are ready; latest deployment marker 622bc372 points runtime image to 2ec7f6f4 and production API health returns healthy. CD #3294 still has a historical Failure record because post-deploy monitoring coverage saw 188 nginx-exporter down before the exporter restore.
|
||||
MOMO state: mo.wooo.work health is healthy on version V10.645; current-month daily_sales_snapshot and realtime_sales_monthly match, but both stop at 2026-06-17. MOMO_DAILY_FRESHNESS is 7 days, which is a hard blocker because business data is not current.
|
||||
Google Drive state: momo scheduler token ownership is fixed for Docker userns, container-side Drive listing works, but folder 當日業績匯入 currently has no matching 即時業績_當日 Excel source file. Archive latest matching file is 2026-06-18T01:30:39Z and was already imported by job 56.
|
||||
Backup / monitoring state: backup-status core blockers are 0, last aggregate is 2026-06-24 02:28:39, 188 MinIO is healthy, Velero BackupStorageLocation default is Available, one-off backup reboot-recovery-202606240456 completed, backup-health textfile reports Velero freshness green, and VeleroBackupNotRun / PostgreSQLDown / RedisDown / disk-pressure alerts resolved.
|
||||
Backup / monitoring state: backup-status core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, last aggregate is 2026-06-24 02:28:39, 188 MinIO is healthy, Velero BackupStorageLocation default is Available, one-off backup reboot-recovery-202606240456 completed, backup-health textfile reports Velero freshness green, PostgreSQL / Redis exporters are green, 188 nginx-exporter is restored with nginx_up=1, monitoring coverage is 14/14 jobs UP, and VeleroBackupNotRun / PostgreSQLDown / RedisDown / disk-pressure / nginx-exporter target-down evidence is resolved.
|
||||
Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice.
|
||||
Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom.
|
||||
Allowed declaration: core hosts, routes, K3s, backup/exporter surfaces are recovered; MOMO data pipeline is blocked waiting for a newer source file or owner-provided source evidence.
|
||||
Forbidden declaration: full-stack green, MOMO data current, DR complete, or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged.
|
||||
```
|
||||
@@ -1931,6 +1932,47 @@ No safe import candidate exists.
|
||||
Full-stack remains blocked by data freshness, not by service outage.
|
||||
```
|
||||
|
||||
### 14.32 2026-06-24 188 nginx-exporter / CD monitoring coverage gate
|
||||
|
||||
2026-06-24 的第六段變更是把 CD post-deploy monitoring coverage 失敗納入重啟 SOP。`2ec7f6f4` 的 runtime deploy 已回寫 `622bc372` 並且 production API health 為 healthy，但 CD `#3294` 的 post-deploy checks 因 `nginx-exporter` target down 留下 Failure。根因是 188 `nginx-exporter` container 未運行，並非 Nginx public gateway、API/Web rollout 或 MOMO 服務故障。
|
||||
|
||||
| 項目 | 20:10 monitoring coverage baseline |
|
||||
|------|------------------------------------|
|
||||
| SOP version | `v1.34` |
|
||||
| Affected CD run | Gitea CD `#3294` 歷史結果仍為 Failure；deploy marker `622bc372` 已寫入 |
|
||||
| Root cause | Prometheus job `nginx-exporter` down，target `192.168.0.188:9113` connection refused |
|
||||
| Non-root cause | Nginx `stub_status` 正常；不需要 reload Nginx、不需要重啟 API/Web/MOMO、不需要改 firewall |
|
||||
| Live restore source | `/home/ollama/nginx-exporter.yml` |
|
||||
| Repo helper | `scripts/ops/188-nginx-exporter-restore.sh` |
|
||||
| Check mode | `--check` only reads stub_status, compose config, container state, and metrics |
|
||||
| Apply mode | `--apply` runs `docker compose -f /home/ollama/nginx-exporter.yml up -d` after stub_status and compose config pass |
|
||||
| Exporter metrics | `nginx_up 1`、`nginx_connections_active`、`nginx_http_requests_total` |
|
||||
| Monitoring coverage | `Jobs 總數=14`、`全部 UP=14`、`真實問題=0`、`預期覆蓋率=100.0%` |
|
||||
| Declaration limit | 可宣稱 exporter / monitoring coverage recovered；不可把歷史 CD run 改稱 success，也不可宣稱 full-stack green / DR complete |
|
||||
|
||||
Post-reboot / post-CD 188 nginx-exporter check:
|
||||
|
||||
```bash
|
||||
bash scripts/ops/188-nginx-exporter-restore.sh --check
|
||||
python3 scripts/generate_monitoring.py --check --stabilization-sleep-seconds 0
|
||||
```
|
||||
|
||||
如果 `--check` 只在 metrics 階段失敗，但 `stub_status` 與 compose config 都通過，且維護窗口允許恢復無狀態 exporter：
|
||||
|
||||
```bash
|
||||
bash scripts/ops/188-nginx-exporter-restore.sh --apply
|
||||
python3 scripts/generate_monitoring.py --check --stabilization-sleep-seconds 0
|
||||
```
|
||||
|
||||
禁止把這個症狀用下列方式處理：
|
||||
|
||||
```text
|
||||
NO-GO: reload Nginx before stub_status / exporter metrics prove Nginx config is the issue.
|
||||
NO-GO: restart product containers because monitoring coverage alone is red.
|
||||
NO-GO: silence monitoring coverage or mark CD green without target recovery evidence.
|
||||
NO-GO: prune Docker volumes or delete exporter state not owned by this SOP.
|
||||
```
|
||||
|
||||
### 14.22 重啟後時間軸驗證
|
||||
|
||||
每次重啟後照時間軸推進，不要等到最後才一次判定。
|
||||
|
||||
Reference in New Issue
Block a user