fix(ops): add gitea pressure check-mode playbook
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Failing after 51s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped

This commit is contained in:
Your Name
2026-07-02 12:51:05 +08:00
parent f41f77cfb7
commit 3f8d784698
9 changed files with 734 additions and 7 deletions

View File

@@ -1,3 +1,30 @@
## 2026-07-02 — 12:50 110 Gitea CPU 壓力 check-mode playbook 實作
**完成內容**
- 新增 `scripts/ops/gitea-queue-hook-backlog-playbook.py`,把 controller 的 `gitea_queue_or_hook_backlog` 從泛用 evidence 分流,升級成 Gitea 專用 read-only check-mode只讀公開 `/api/healthz``/api/v1/version``/metrics` 與 sanitized host textfile不讀 app.ini / secret / `.runner` / raw session / DB。
- `scripts/ops/host-sustained-load-controller.py` 的兩條 Gitea 壓力分支現在輸出 `/home/wooo/scripts/gitea-queue-hook-backlog-playbook.py ... --json`StockPlatform、control-plane、unknown 分支仍維持 `host-sustained-load-evidence.py`
- `.gitea/workflows/cd.yaml``ops/runner/test_cd_controlled_runtime_profile.py` 已把新 playbook / test 納入 controlled-runtime allowlist、`py_compile` 與 focused pytest避免 P0 ops 修法誤跑 B5 Docker socket。
- 已 live 部署到 110`/home/wooo/scripts/gitea-queue-hook-backlog-playbook.py` SHA `361197681f23482a172ee2de9170b5669f58ac089b1d7ca3f2a18300bed3059c``/home/wooo/scripts/host-sustained-load-controller.py` SHA `96bb93e8169af5e98aac18d123daec89b9d8a74ce194abea16f5a75e051f4f64`
**live readback 證據**
- controller 現在回報 `classification=blocked_gitea_queue_or_hook_backlog_requires_playbook`,且 `dry_run=/home/wooo/scripts/gitea-queue-hook-backlog-playbook.py ... --json`
- Gitea playbook 回報 `classification=blocked_gitea_hooktask_backlog_check_required``gitea_health=status pass``version=1.25.5``active_actions container_count=0 process_group_count=0``docker_stats fresh=true age=15s``gitea_container_cpu_cores=1.6166``gitea_process_family_cpu_percent=53.1``gitea_hooktasks=1218``go_goroutines=159``process_open_fds=88`
- operation boundaries 全部保持 read-only`host_write_performed=false``process_signal_performed=false``docker_restart_performed=false``systemd_restart_performed=false``database_query_performed=false``secret_value_read=false`
**驗證**
- `python3.11 -m pytest scripts/ops/tests/test_gitea_queue_hook_backlog_playbook.py scripts/ops/tests/test_host_runaway_process_exporter.py scripts/ops/tests/test_host_pressure_alert_contract.py ops/runner/test_cd_controlled_runtime_profile.py -q``75 passed`
- `python3.11 -m py_compile scripts/ops/gitea-queue-hook-backlog-playbook.py scripts/ops/host-sustained-load-controller.py`:通過。
- `.gitea/workflows/cd.yaml` / `ops/monitoring/alerts-unified.yml` YAML parse通過。
- `python3.11 ops/runner/guard-gitea-runner-pressure.py --root .`:通過,`auto_branch_events_on_110=0``generic_runner_labels=0`
- `git diff --check`:通過。
**仍維持**
- 未使用 GitHub / `gh` / GitHub API未讀 secret / token / `.env` / raw sessions / SQLite / auth未讀 `.runner` 內容。
- 未重啟主機,未 restart Docker / Nginx / K3s / DB / firewall未 workflow_dispatch未 DROP / TRUNCATE / restore / prune。
**下一步**
- commit / push Gitea `main`,讀回 CDCD 成功後下一個 P0 是把 `blocked_gitea_hooktask_backlog_check_required` 接成 public queue readback / authorized export-free 判讀,避免下次又只停在「知道 Gitea 熱」。
## 2026-07-02 — 11:28 110 cold-start SSH 探測誤判修正
**完成內容**

View File

@@ -96,6 +96,8 @@ v1.82 bounded summary rule`post-start-quick-check.sh` 與 `188-host-hygiene-m
2026-07-02 12:35 追加 110 CPU 分流優先序:若 Docker stats 是 fresh`gitea` 或 StockPlatform 關鍵容器已超過 early triage 門檻 `1.0 core`controller / evidence 必須先路由到對應服務 playbook不得被長壽命 `ps %CPU``systemd_control_plane` 平均值搶先導到 control-plane playbook。control-plane saturation 仍保留為後備路徑,適用於沒有已知 hot container / hot service family 的情境。
2026-07-02 12:50 追加 Gitea 壓力 check-mode`host-sustained-load-controller.py` 若分類為 `blocked_gitea_queue_or_hook_backlog_requires_playbook``dry_run_command` 必須指向 110 已部署的 `/home/wooo/scripts/gitea-queue-hook-backlog-playbook.py --host 110 --metrics-file /home/wooo/node_exporter_textfiles/host_runaway_process.prom --docker-stats-file /home/wooo/node_exporter_textfiles/docker_stats.prom --json`。此 playbook 只能讀公開 Gitea health/version/metrics 與 sanitized host textfile必須輸出 health、version、active Actions、fresh Docker stats、`gitea_hooktasks`、Gitea container CPU、process-family CPU 與 operation boundaries。若 health/version 正常、active Actions 為 0、`gitea` container 超過 `1.0 core``gitea_hooktasks >= 1000`,狀態固定為 `blocked_gitea_hooktask_backlog_check_required`,下一步走 public queue / hook backlog 判讀;不得讀 app.ini / secret / `.runner` / raw session / DB不得重啟 Gitea / Docker / Nginx / K3s / DB / firewall。
2026-06-25 20:25 orphan Chrome cleanup / scorecard refresh supersedes the 20:11 CPU wording. 110 high CPU was traced to two `stockplatform-review-bulk-ux` Chrome process groups `2756503` and `2829627` with root Chrome process `PPID=1`, elapsed about 5h, no active parent smoke, and sustained GPU/renderer CPU. With user approval, only those two process groups received targeted `SIGTERM` at 20:24. Post-check showed no remaining PGID entries; `vmstat` showed CPU idle around `85-90%`, `si/so=0`, and no immediate swap thrash. No Docker/systemd/Nginx/firewall/K8s action, CI cancellation, manual data ingestion, manual DB write, Wazuh/SOC runtime change, or secret read was performed. The 20:25 full post-start wrapper then returned cold-start `PASS=89 WARN=0 BLOCKED=0`, but overall `POST_START_QUICK_CHECK PASS=37 WARN=2 BLOCKED=1`, `RESULT=BLOCKED`, because StockPlatform data freshness was still blocked at that time and DR remained incomplete.
2026-06-25 20:11 StockPlatform cron-source recovery supersedes the 19:35 source-version wording. StockPlatform Gitea `main` and live `/home/wooo/stockplatform-v2` are now at `fb91aa4c6272469d1d26e0820169629eac17d28a fix(ops): restore production cron recovery entrypoints`; six missing production cron entrypoint scripts are restored, `run-intelligence-sync.sh` contains the Docker-backed `psql` shim, and live contract check confirms every `scripts/ops/*.sh` referenced by `install-production-cron.sh` exists. The only live write performed for StockPlatform recovery was a fast-forward `git pull --ff-only origin main` on 110; no Docker/systemd/Nginx/firewall/K8s restart, manual ingestion run, manual DB write, or secret read was performed. Natural cron evidence after the pull is now green for the repaired entrypoints: `source-remediation-queue` 19:56 and 20:00 succeeded, `market-index-ingestion` 20:00 succeeded, `price-ingestion` 20:02 succeeded, `margin-short-ingestion` 20:05 succeeded, `chips-ingestion` 20:06 succeeded, and `ai-recommendation-pipeline` 20:10 ran but correctly produced the internal blocker `core_margin_short_daily_incomplete,official_margin_short_daily_official_pending`. StockPlatform `/api/v1/system/freshness` therefore still returns `status=blocked` because the 2026-06-25 official margin-short source is pending and `ai.recommendations` must stay on 2026-06-24 until that gate clears. This is no longer a route, source-version, or missing-cron-script blocker; it is a product-data freshness blocker waiting on official source availability and the next valid AI pipeline run.

View File

@@ -1,7 +1,7 @@
# 主機異常行程 AIOps PlayBook
> 最後更新2026-06-18 Asia/Taipei
> 範圍110 host CPU 滿載、orphan Chrome / Playwright smoke、Gitea Actions CI load、Prisma / package install 資源壓力分流。
> 最後更新2026-07-02 Asia/Taipei
> 範圍110 host CPU 滿載、orphan Chrome / Playwright smoke、Gitea Actions CI load、Gitea 服務熱點、Prisma / package install 資源壓力分流。
---
@@ -19,6 +19,7 @@ read-only exporter -> Prometheus alert -> AI triage packet -> KM / PlayBook evid
|------|------|------|
| orphan browser smoke | headless Chrome / Chromium / Playwright process group 存活過久、PPID=1 或 group leader 消失、CPU 合計過高 | 走 dry-run 修復包controlled apply receipt + evidence + verifier 成立後可送 `SIGTERM` |
| 合法 CI load | Gitea Actions task container 正在跑,沒有 orphan browser 指標 | 觀察 queue / timeout不要誤殺 |
| Gitea 服務熱點 | Docker stats fresh`gitea` container 超過 early triage 門檻,且 active Actions 為 0 | 走 `gitea-queue-hook-backlog-playbook.py` check-mode先讀 public health/version/metrics 與 sanitized host textfile不重啟服務 |
| Docker / Sentry / Harbor 事故 | container restart、port down、journal error、cold-start gate blocked | 走各服務自己的 SOP不使用本 PlayBook 殺 process |
| swap 已滿但未 thrash | swap ratio 高但 `vmstat` / load 分類未顯示即時 thrash | 不手動清 swap先降高 CPU 來源 |
@@ -31,6 +32,7 @@ read-only exporter -> Prometheus alert -> AI triage packet -> KM / PlayBook evid
```text
/home/wooo/scripts/host-runaway-process-exporter.py
/home/wooo/scripts/host-runaway-process-remediation.py
/home/wooo/scripts/gitea-queue-hook-backlog-playbook.py
/home/wooo/node_exporter_textfiles/host_runaway_process.prom
```
@@ -42,6 +44,7 @@ read-only exporter -> Prometheus alert -> AI triage packet -> KM / PlayBook evid
| `awoooi_host_runaway_browser_orphan_group_count{host="110",rule=...}` | 符合規則的 orphan browser process group 數 |
| `awoooi_host_runaway_browser_orphan_cpu_percent{host="110",rule=...}` | orphan group CPU 合計 |
| `awoooi_host_gitea_actions_active_container_count{host="110"}` | 目前 active Gitea Actions task containers |
| `awoooi_host_process_family_cpu_percent{host="110",family="gitea_service"}` | Gitea process family CPU用來補足 Docker stats 歸因 |
| `awoooi_host_load5_per_core{host="110"}` | load5 / CPU core |
| `awoooi_host_swap_used_ratio{host="110"}` | swap 使用比例 |
| `awoooi_host_runaway_process_remediation_authorized{host="110"}` | 必須永遠為 `0`exporter 不是執行器 |
@@ -93,6 +96,18 @@ dry-run 必須檢查:
如果只看到 `HostCiRunnerLoadSaturation`,且 orphan group count 為 `0`,預設判定是「合法 CI 短期負載」,不得自動殺 process只能走 runner queue verifier、stale-run drain/cancel packet 與 host pressure fail-closed。
如果 controller 回報 `blocked_gitea_queue_or_hook_backlog_requires_playbook`,下一步固定跑 Gitea 專用 check-mode
```bash
python3 scripts/ops/gitea-queue-hook-backlog-playbook.py \
--host 110 \
--metrics-file /home/wooo/node_exporter_textfiles/host_runaway_process.prom \
--docker-stats-file /home/wooo/node_exporter_textfiles/docker_stats.prom \
--json
```
Gitea playbook 只能讀公開 `/api/healthz``/api/v1/version``/metrics` 與 sanitized host textfile。必須輸出 active Actions、Gitea health/version、`gitea_hooktasks`、Gitea container CPU、process-family CPU、Docker stats freshness 與 operation boundaries不得讀 app.ini、secret、`.runner`、raw session、DB也不得 restart / reload / kill。若 health 正常、active Actions 為 0、Gitea container 超過 `1.0 core``gitea_hooktasks >= 1000`,分類為 `blocked_gitea_hooktask_backlog_check_required`,再進 public queue / hook backlog 判讀。
如果只看到 `HostLoadAverageSustainedHigh`,且 orphan / active CI / swap 都無明確命中AI 必須先跑只讀脫敏 evidence collector
```bash