fix(ops): classify hostedtoolcache ci load [metadata-only]
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Failing after 1m56s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped

This commit is contained in:
Your Name
2026-07-02 00:33:42 +08:00
parent 695e887e68
commit c7241365eb
6 changed files with 91 additions and 5 deletions

View File

@@ -51990,6 +51990,28 @@ production browser smoke:
**下一步**
- 以 `[skip ci]` source commit 推回 Gitea main避免只剩 live drift然後回到 Gitea queue / 110 control-path / reboot SLO blocker 主線。
## 2026-07-02 — 00:31 P0 110 active CI load classifier 修正
**完成內容**
- `HostLoadAverageSustainedHigh` 再次 firing 時SSH 診斷讀回 `node /opt/hostedtoolcache/node/20.20.2/x64/bin/pnpm install --frozen-lockfile`CPU 約 `269%`;這是 Gitea Actions 類 CI install load不是 StockPlatform DB、orphan browser、DB deadlock 或磁碟 IO wait。
- 同輪 `vmstat` 顯示 `b=0`、IO wait 約 `0-1%`,所以不是 disk stuckStockPlatform DB 已被 cgroup 壓住,後續 direct stats 回到低 CPU。
- 根因是 `host-runaway-process-exporter.py` / `host-sustained-load-evidence.py` 未把 `/opt/hostedtoolcache``pnpm install --frozen-lockfile``npm ci``yarn install` 歸類為 active Gitea Actions load導致 `awoooi_host_gitea_actions_active_process_count` 可誤報 `0`
- 已修正 classifieractive CI install load 會走 runner / queue / pressure gate lane不會再誤導成 unknown / Gitea backlog / Stock DB 壓力。
- 新增 regression test確認 sanitized evidence 不輸出 `/opt/hostedtoolcache` 或 raw `pnpm install` command。
**本地驗證結果**
- `python3.11 -m pytest scripts/ops/tests/test_host_runaway_process_exporter.py scripts/ops/tests/test_host_pressure_alert_contract.py -q``24 passed`
- `python3.11 -m py_compile scripts/ops/host-runaway-process-exporter.py scripts/ops/host-sustained-load-evidence.py scripts/ops/tests/test_host_runaway_process_exporter.py`:通過。
- `git diff --check`:通過。
**仍維持**
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth。
- 沒有使用 GitHub / gh / GitHub API / GitHub Actions。
- 沒有重啟主機,沒有 Docker / Nginx / K3s / DB / firewall restart沒有 workflow_dispatch沒有 DROP / TRUNCATE / restore / prune / process kill。
**下一步**
- 同步新版 exporter / evidence script 到 110 live textfile path跑一次 read-only exporter讀回 active CI metrics再 rebase / commit / push source。
## 2026-07-02 — P0 AI loop stale control-path blocker production readback 修正
**完成內容**

View File

@@ -147,7 +147,10 @@
},
"source_updates": [
"scripts/ops/docker-disk-pressure-retention-cleanup.py",
"scripts/ops/host-runaway-process-exporter.py",
"scripts/ops/host-sustained-load-evidence.py",
"scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py",
"scripts/ops/tests/test_host_runaway_process_exporter.py",
"ops/reboot-recovery/full-stack-cold-start-baseline.yml",
"docs/runbooks/FULL-STACK-COLD-START-SOP.md",
"docs/LOGBOOK.md"

View File

@@ -2499,3 +2499,25 @@ docker update --cpus=1.5 --memory=4g --memory-swap=6g --restart unless-stopped s
- 可宣稱110 sustained moderate CPU alert 已主動觸發、已定位、已用 Docker guardrail 降壓StockPlatform route / health / freshness 未受損。
- 不可宣稱110 永久不會再高負載Gitea queue / hook backlog 仍需在後續 queue / control-path lane 持續觀察。
- 不得用 DB restart、container restart、manual DB write、query kill 或 fake freshness 來處理 StockPlatform pressure。
### 16.5 2026-07-02 hostedtoolcache / pnpm install 必須算 active CI load
同一輪 110 high load 診斷發現 `node /opt/hostedtoolcache/node/.../pnpm install --frozen-lockfile` 會造成短期 runnable burst。這是 Gitea Actions 類 CI install load但舊 exporter 只認 `/.cache/act/`、Docker build、BuildKit、`next build`,導致 `awoooi_host_gitea_actions_active_process_count` 誤報 `0`,進而讓 controller 看起來像 unknown / Gitea backlog而不是 active CI。
修正 source-of-truth
- `scripts/ops/host-runaway-process-exporter.py`
- `scripts/ops/host-sustained-load-evidence.py`
- `scripts/ops/tests/test_host_runaway_process_exporter.py`
新增分類規則:
- `/opt/hostedtoolcache/`
- `pnpm install`
- `npm ci`
- `yarn install`
宣告限制:
- active CI load 只能走 runner / queue / pressure gate lane不得直接 `SIGKILL``docker restart``systemctl restart`、DB restart 或恢復 generic runner。
- evidence output 仍必須保持 sanitized不輸出 hostedtoolcache 完整路徑、raw command、URL、workspace path 或 secret。