wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

fix(ops): expose host runner build load in exporter [skip ci]

Browse Source
This commit is contained in:
ogt
2026-06-27 12:57:53 +08:00
parent b623dc6011
commit fcd4337b3a
4 changed files with 110 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
docs/runbooks/HOST-RESOURCE-BASELINE-110-188.md
View File

@@ -52,7 +52,7 @@ Use these thresholds for alerting and AI triage:
| Systemd runner restarts | > 2 in 15m | Critical; inspect watchdog/drop-ins and active CI jobs. |
| Systemd runner WatchdogSec | > 0 for 10m | Warning; GitHub Actions runner should not be killed by systemd watchdog. |
| Systemd runner quota | CPU or memory unlimited for 30m | Warning; apply CPUQuota/MemoryMax or move CI away from Sentry host. |
| Gitea Actions job runtime | > 20m for 5m | Warning; inspect logs and run `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh` dry-run before stopping stale job containers. The stale-job detector must include both legacy `GITEA-ACTIONS-*` containers and repo-scoped names such as `awoooi-cd-<run>-<job>` / `awoooi-code-review-<run>-<job>`. |
| Gitea Actions job runtime | > 20m for 5m | Warning; inspect logs and run `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh` dry-run before stopping stale job containers. The stale-job detector must include both legacy `GITEA-ACTIONS-*` containers and repo-scoped names such as `awoooi-cd-<run>-<job>` / `awoooi-code-review-<run>-<job>`。textfile exporter 也必須輸出 host-side `act_runner` / BuildKit / `next build` process load，因為活躍 CD build 可能不會出現在具名 action container 裡。 |
## Rules
@@ -67,8 +67,9 @@ Use these thresholds for alerting and AI triage:
9. Disable node-exporter collectors that are slow or failing on each host; exporter scrape time is part of the resource baseline.
10. Runner services are part of the resource baseline even when jobs run outside Docker. Export `systemd_unit_*` metrics for all `actions.runner.*` services.
11. CI runner limits must be set at the systemd unit level. Docker container limits do not protect host load from runner listener, worker, and `act` helper processes.
12. Workflow-only commits must not trigger full CD image builds; CD should run only when runtime inputs change.
13. Stale Gitea Actions cleanup must be dry-run first. Use `--apply` only when the container is older than its workflow/job policy threshold and has no recent useful logs.
12. 當 `docker ps` 沒有 action container 時，優先用 `awoooi_host_gitea_actions_active_process_*` 判讀 110 高 CPU；BuildKit / `runc` / `next build` 可能仍在跑，即使 `awoooi_host_gitea_actions_active_container_count=0`。
13. Workflow-only commits must not trigger full CD image builds; CD should run only when runtime inputs change.
14. Stale Gitea Actions cleanup must be dry-run first. Use `--apply` only when the container is older than its workflow/job policy threshold and has no recent useful logs.
## Next Safe Rollout Order
@@ -79,7 +80,7 @@ Use these thresholds for alerting and AI triage:
5. Tune `momo-scheduler` crawler concurrency on 188; keep 2 CPU / 2 GiB until success rate and latency prove it is too low.
6. Fix 188 Elephant Alpha/OpenClaw allowed-action drift before enabling resource auto-repair beyond diagnosis.
7. Add modest caps to currently unlimited low-risk services in small batches. Do not alert every unlimited auxiliary container at once; promote candidates only after 24h usage data.
8. Deploy `scripts/ops/stop-stale-gitea-actions-jobs.sh` to 110 as `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh`; keep Prometheus auto action in dry-run mode. As of 2026-06-27, it must recognize both legacy `GITEA-ACTIONS-*` and current repo-scoped `*-cd-<run>-*` / `*-code-review-<run>-*` container names.
8. Deploy `scripts/ops/stop-stale-gitea-actions-jobs.sh` to 110 as `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh`; keep Prometheus auto action in dry-run mode. As of 2026-06-27, it must recognize both legacy `GITEA-ACTIONS-*` and current repo-scoped `*-cd-<run>-*` / `*-code-review-<run>-*` container names；同時 `host-runaway-process-exporter.py` 必須在沒有 action container 可見時仍輸出 BuildKit / `next build` process load。
9. Fix 110 runner services with sudo-capable host maintenance:
```bash