fix(ops): classify hostedtoolcache ci load [metadata-only]
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Failing after 1m56s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Failing after 1m56s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped
This commit is contained in:
@@ -51990,6 +51990,28 @@ production browser smoke:
|
||||
**下一步**:
|
||||
- 以 `[skip ci]` source commit 推回 Gitea main,避免只剩 live drift;然後回到 Gitea queue / 110 control-path / reboot SLO blocker 主線。
|
||||
|
||||
## 2026-07-02 — 00:31 P0 110 active CI load classifier 修正
|
||||
|
||||
**完成內容**:
|
||||
- `HostLoadAverageSustainedHigh` 再次 firing 時,SSH 診斷讀回 `node /opt/hostedtoolcache/node/20.20.2/x64/bin/pnpm install --frozen-lockfile`,CPU 約 `269%`;這是 Gitea Actions 類 CI install load,不是 StockPlatform DB、orphan browser、DB deadlock 或磁碟 IO wait。
|
||||
- 同輪 `vmstat` 顯示 `b=0`、IO wait 約 `0-1%`,所以不是 disk stuck;StockPlatform DB 已被 cgroup 壓住,後續 direct stats 回到低 CPU。
|
||||
- 根因是 `host-runaway-process-exporter.py` / `host-sustained-load-evidence.py` 未把 `/opt/hostedtoolcache`、`pnpm install --frozen-lockfile`、`npm ci`、`yarn install` 歸類為 active Gitea Actions load,導致 `awoooi_host_gitea_actions_active_process_count` 可誤報 `0`。
|
||||
- 已修正 classifier,active CI install load 會走 runner / queue / pressure gate lane,不會再誤導成 unknown / Gitea backlog / Stock DB 壓力。
|
||||
- 新增 regression test,確認 sanitized evidence 不輸出 `/opt/hostedtoolcache` 或 raw `pnpm install` command。
|
||||
|
||||
**本地驗證結果**:
|
||||
- `python3.11 -m pytest scripts/ops/tests/test_host_runaway_process_exporter.py scripts/ops/tests/test_host_pressure_alert_contract.py -q`:`24 passed`。
|
||||
- `python3.11 -m py_compile scripts/ops/host-runaway-process-exporter.py scripts/ops/host-sustained-load-evidence.py scripts/ops/tests/test_host_runaway_process_exporter.py`:通過。
|
||||
- `git diff --check`:通過。
|
||||
|
||||
**仍維持**:
|
||||
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth。
|
||||
- 沒有使用 GitHub / gh / GitHub API / GitHub Actions。
|
||||
- 沒有重啟主機,沒有 Docker / Nginx / K3s / DB / firewall restart,沒有 workflow_dispatch,沒有 DROP / TRUNCATE / restore / prune / process kill。
|
||||
|
||||
**下一步**:
|
||||
- 同步新版 exporter / evidence script 到 110 live textfile path,跑一次 read-only exporter,讀回 active CI metrics;再 rebase / commit / push source。
|
||||
|
||||
## 2026-07-02 — P0 AI loop stale control-path blocker production readback 修正
|
||||
|
||||
**完成內容**:
|
||||
|
||||
@@ -147,7 +147,10 @@
|
||||
},
|
||||
"source_updates": [
|
||||
"scripts/ops/docker-disk-pressure-retention-cleanup.py",
|
||||
"scripts/ops/host-runaway-process-exporter.py",
|
||||
"scripts/ops/host-sustained-load-evidence.py",
|
||||
"scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py",
|
||||
"scripts/ops/tests/test_host_runaway_process_exporter.py",
|
||||
"ops/reboot-recovery/full-stack-cold-start-baseline.yml",
|
||||
"docs/runbooks/FULL-STACK-COLD-START-SOP.md",
|
||||
"docs/LOGBOOK.md"
|
||||
|
||||
@@ -2499,3 +2499,25 @@ docker update --cpus=1.5 --memory=4g --memory-swap=6g --restart unless-stopped s
|
||||
- 可宣稱:110 sustained moderate CPU alert 已主動觸發、已定位、已用 Docker guardrail 降壓,StockPlatform route / health / freshness 未受損。
|
||||
- 不可宣稱:110 永久不會再高負載;Gitea queue / hook backlog 仍需在後續 queue / control-path lane 持續觀察。
|
||||
- 不得用 DB restart、container restart、manual DB write、query kill 或 fake freshness 來處理 StockPlatform pressure。
|
||||
|
||||
### 16.5 2026-07-02 hostedtoolcache / pnpm install 必須算 active CI load
|
||||
|
||||
同一輪 110 high load 診斷發現 `node /opt/hostedtoolcache/node/.../pnpm install --frozen-lockfile` 會造成短期 runnable burst。這是 Gitea Actions 類 CI install load,但舊 exporter 只認 `/.cache/act/`、Docker build、BuildKit、`next build`,導致 `awoooi_host_gitea_actions_active_process_count` 誤報 `0`,進而讓 controller 看起來像 unknown / Gitea backlog,而不是 active CI。
|
||||
|
||||
修正 source-of-truth:
|
||||
|
||||
- `scripts/ops/host-runaway-process-exporter.py`
|
||||
- `scripts/ops/host-sustained-load-evidence.py`
|
||||
- `scripts/ops/tests/test_host_runaway_process_exporter.py`
|
||||
|
||||
新增分類規則:
|
||||
|
||||
- `/opt/hostedtoolcache/`
|
||||
- `pnpm install`
|
||||
- `npm ci`
|
||||
- `yarn install`
|
||||
|
||||
宣告限制:
|
||||
|
||||
- active CI load 只能走 runner / queue / pressure gate lane;不得直接 `SIGKILL`、`docker restart`、`systemctl restart`、DB restart 或恢復 generic runner。
|
||||
- evidence output 仍必須保持 sanitized,不輸出 hostedtoolcache 完整路徑、raw command、URL、workspace path 或 secret。
|
||||
|
||||
Reference in New Issue
Block a user