fix(ops): use deployed host pressure helper paths
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 1s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 1m3s
CD Pipeline / build-and-deploy (push) Successful in 4m58s
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 1s
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Successful in 10s
CD Pipeline / post-deploy-checks (push) Successful in 3m11s

This commit is contained in:
Your Name
2026-07-02 12:14:35 +08:00
parent 36f1fffbda
commit 859e407129
4 changed files with 49 additions and 24 deletions

View File

@@ -52440,21 +52440,23 @@ production browser smoke:
- `scripts/ops/host-runaway-process-exporter.py` 新增 sanitized process-family metrics`awoooi_host_process_family_cpu_percent``process_count``oldest_age_seconds``top_info`;只輸出 `family` / `comm`,不輸出 raw command line、workspace path、URL 或 secret。
- `scripts/ops/host-sustained-load-controller.py` 新增 live `ps` process-family 分類,並把中度壓力分流到 `control_plane_saturation``gitea_queue_or_hook_backlog``stockplatform_hot_query_or_api_pressure` check-mode packetcontroller 仍只產生 controlled packet不做 Docker / systemd / DB / Nginx restart。
- `ops/monitoring/alerts-unified.yml``Host110SustainedModeratePressure` 從只看 `load5/core` / container CPU擴充為同時看 `awoooi_host_process_family_cpu_percent{family=~"systemd_control_plane|ssh_control_plane|gitea_service|postgres"} > 50`Gitea / StockPlatform container early triage 門檻從 `> 2.0 core` 降到 `> 1.0 core`
- `scripts/ops/host-sustained-load-controller.py` 追加 `--script-dir`,預設 `/home/wooo/scripts`,確保 live alert 產出的 `dry_run_command` / `post_apply_verifier` 指向 110 實際部署 helper不再輸出會打空的 repo 相對路徑 `scripts/ops/...`
- 已 live 部署到 110
- `/home/wooo/scripts/host-runaway-process-exporter.py` SHA `d85d27c81ea76a8f2f370ee85c92381bec4440eea4fd37efb2efb9f43dbd1a8a`
- `/home/wooo/scripts/host-sustained-load-controller.py` SHA `c731c6b5fe5e1683931cf949b3b18c8b290c9a5d5973fac826487ad8de05434a`
- `/home/wooo/scripts/host-sustained-load-controller.py` SHA `7a11407c7df05427085982d6f6d11d1756f908591573a28c9b3267de32b94f3e`
- `bash scripts/ops/deploy-alerts.sh` 完成Prometheus 已載入 `159` 條規則。
**live readback 證據**
- 110 textfile 已輸出 process-family metric`systemd_control_plane=72.4``gitea_service=53.2``postgres=23.3`
- 110 textfile / Prometheus 已輸出 process-family metric`systemd_control_plane=72.4``gitea_service=53.1``postgres=11.1`
- Prometheus rule readback`Host110SustainedModeratePressure state=firing health=ok`query 已包含 `docker_container_cpu_cores > 1``awoooi_host_process_family_cpu_percent > 50`
- Prometheus `ALERTS{alertname="Host110SustainedModeratePressure"}` 已有 `firing``container_name="gitea"``family="systemd_control_plane"``family="gitea_service"``stockplatform-v2-postgres-1` 在部署後短時間為 `pending`
- Alertmanager `/api/v2/alerts` 已有 active alert`status.state=active``notification_type=TYPE-1``auto_repair=true``team=ops`
- live controller readback`classification=blocked_control_plane_saturation_requires_playbook``next_action=run_control_plane_saturation_playbook_check_mode``control_plane_process_cpu_percent=72.4``gitea_process_cpu_percent=53.2``top_container_cpu.gitea=0.2507~1.5789`,且 `docker_stats.fresh=true`
- Alertmanager `/api/v2/alerts` 已有 active alert`family=gitea_service``family=systemd_control_plane``status.state=active``auto_repair=true`
- live controller readback`classification=blocked_control_plane_saturation_requires_playbook``next_action=run_control_plane_saturation_playbook_check_mode``dry_run_command=/home/wooo/scripts/host-sustained-load-evidence.py ...``post_apply_verifier=/home/wooo/scripts/host-sustained-load-controller.py ...``controller_exit=75`
- live sanitized evidence readback`recommendation=control_plane_saturation_playbook``controlled_apply_allowed=false``top_process_families` 包含 `systemd_control_plane=72.4``gitea_service=53.1``unknown=54.3``top_containers``gitea=1.5951 core` 為最高evidence 明確標示不輸出 raw command line / URL / secret
**本地驗證結果**
- `python3.11 -m py_compile scripts/ops/host-runaway-process-exporter.py scripts/ops/host-sustained-load-controller.py`:通過。
- `python3.11 -m py_compile scripts/ops/host-runaway-process-exporter.py scripts/ops/host-sustained-load-controller.py scripts/ops/host-sustained-load-evidence.py`:通過。
- `python3.11 -m pytest scripts/ops/tests/test_host_runaway_process_exporter.py scripts/ops/tests/test_host_pressure_alert_contract.py -q``27 passed`
- `python3.11 -m pytest ops/runner/test_cd_controlled_runtime_profile.py scripts/ops/tests/test_host_runaway_process_exporter.py scripts/ops/tests/test_host_pressure_alert_contract.py -q``70 passed`
- `python3.11 -c "import yaml; yaml.safe_load(open('ops/monitoring/alerts-unified.yml'))"`:通過。
- `python3.11 ops/runner/guard-gitea-runner-pressure.py --root .`:通過,`auto_branch_events_on_110=0``generic_runner_labels=0`
- `git diff --check`:通過。

View File

@@ -92,6 +92,8 @@ v1.82 bounded summary rule`post-start-quick-check.sh` 與 `188-host-hygiene-m
2026-07-01 23:28 追加 110 主動告警與 controlled quota 收斂2026-07-02 00:40 更新 Gitea cap2026-07-02 11:54 追加 process-family 主動告警:`HostLoadAverageSustainedHigh``load5/core > 1.5 for 15m` 只能當 critical 門檻不足以回答「110 CPU 又持續偏高為何沒告警」。Prometheus 必須同時有 `Host110SustainedModeratePressure`,在 `awoooi_host_load5_per_core{host="110"} > 0.75`、Gitea / StockPlatform 關鍵容器 `docker_container_cpu_cores > 1.0`,或 `awoooi_host_process_family_cpu_percent{family=~"systemd_control_plane|ssh_control_plane|gitea_service|postgres"} > 50` 持續 1 分鐘時告警auto-repair action 必須指向 110 已部署的 `/home/wooo/scripts/host-sustained-load-controller.py --load5-per-core-threshold 0.75 --hot-container-cpu-threshold 1.0 --process-family-cpu-threshold 50``host-runaway-process-exporter.py` 必須輸出 sanitized process-family metrics只允許 family / comm labels不得輸出 raw command line、workspace path、URL 或 secret。Gitea runtime CPU quota 已從 `docker update --cpus=2 gitea` 進一步收斂為 `docker update --cpus=1.5 gitea`rollback 只允許暫回 `docker update --cpus=2 gitea` 並需 Prometheus readback不可重啟 Docker daemon 或 Gitea container。若 `BackupAggregateRunFailed` 只因 `exported_job="backup_all"` 舊 aggregate failed_count firing而 component jobs / `backup-status.sh` 已 green必須視為噪音並用 component failed count 判斷,不得讓它干擾 cold-start / Telegram 主線。
2026-07-02 12:15 追加 controller command path 合約:`host-sustained-load-controller.py` 產出的 `dry_run_command` / `post_apply_verifier` 必須使用 110 實際部署路徑 `/home/wooo/scripts/host-sustained-load-evidence.py``/home/wooo/scripts/host-sustained-load-controller.py``/home/wooo/scripts/host-runaway-process-remediation.py`。若 controller 在 live host 上輸出 `scripts/ops/...` 這類 repo 相對路徑,視為告警自動化斷鏈,需先修 controller / `--script-dir`,再進入 playbook check-mode不得把「helper 找不到」當成 CPU 根因已處理。
2026-06-25 20:25 orphan Chrome cleanup / scorecard refresh supersedes the 20:11 CPU wording. 110 high CPU was traced to two `stockplatform-review-bulk-ux` Chrome process groups `2756503` and `2829627` with root Chrome process `PPID=1`, elapsed about 5h, no active parent smoke, and sustained GPU/renderer CPU. With user approval, only those two process groups received targeted `SIGTERM` at 20:24. Post-check showed no remaining PGID entries; `vmstat` showed CPU idle around `85-90%`, `si/so=0`, and no immediate swap thrash. No Docker/systemd/Nginx/firewall/K8s action, CI cancellation, manual data ingestion, manual DB write, Wazuh/SOC runtime change, or secret read was performed. The 20:25 full post-start wrapper then returned cold-start `PASS=89 WARN=0 BLOCKED=0`, but overall `POST_START_QUICK_CHECK PASS=37 WARN=2 BLOCKED=1`, `RESULT=BLOCKED`, because StockPlatform data freshness was still blocked at that time and DR remained incomplete.
2026-06-25 20:11 StockPlatform cron-source recovery supersedes the 19:35 source-version wording. StockPlatform Gitea `main` and live `/home/wooo/stockplatform-v2` are now at `fb91aa4c6272469d1d26e0820169629eac17d28a fix(ops): restore production cron recovery entrypoints`; six missing production cron entrypoint scripts are restored, `run-intelligence-sync.sh` contains the Docker-backed `psql` shim, and live contract check confirms every `scripts/ops/*.sh` referenced by `install-production-cron.sh` exists. The only live write performed for StockPlatform recovery was a fast-forward `git pull --ff-only origin main` on 110; no Docker/systemd/Nginx/firewall/K8s restart, manual ingestion run, manual DB write, or secret read was performed. Natural cron evidence after the pull is now green for the repaired entrypoints: `source-remediation-queue` 19:56 and 20:00 succeeded, `market-index-ingestion` 20:00 succeeded, `price-ingestion` 20:02 succeeded, `margin-short-ingestion` 20:05 succeeded, `chips-ingestion` 20:06 succeeded, and `ai-recommendation-pipeline` 20:10 ran but correctly produced the internal blocker `core_margin_short_daily_incomplete,official_margin_short_daily_official_pending`. StockPlatform `/api/v1/system/freshness` therefore still returns `status=blocked` because the 2026-06-25 official margin-short source is pending and `ai.recommendations` must stay on 2026-06-24 until that gate clears. This is no longer a route, source-version, or missing-cron-script blocker; it is a product-data freshness blocker waiting on official source availability and the next valid AI pipeline run.