99 lines
8.7 KiB
Markdown
99 lines
8.7 KiB
Markdown
# 110/188 Host Resource Baseline
|
||
|
||
> 2026-05-05 ogt + Codex
|
||
> Scope: 110 DevOps/Sentry host and 188 AI/Web host Docker Compose resource limits.
|
||
|
||
## Current Live Finding
|
||
|
||
### 110
|
||
|
||
| Service | Live Limit | Live Usage Snapshot | Verdict |
|
||
|---|---:|---:|---|
|
||
| Sentry ClickHouse | 2 CPU / 8 GiB, merge pool 4 | capped near 2 cores after pool 8 -> 4 restart | Do not lower memory. CPU quota intentionally slows background merge so Sentry cannot dominate 110. If backlog grows, inspect `MergeMutate` and Sentry high-volume features before raising it. |
|
||
| Sentry Kafka | 2 CPU / 3 GiB | ~40-55% CPU / 2.5 GiB (84%) | Memory is close to pressure. Do not reduce memory. |
|
||
| Sentry taskworker | 2 CPU / 2 GiB, concurrency 2 | ~120-181% CPU after restart | Concurrency reduced from 4 to 2 after Kafka lag cleared. Watch Sentry task latency before further changes. |
|
||
| Sentry taskbroker | 1 CPU / 512 MiB | ~70-98% CPU / 160 MiB | CPU is tight; increasing may improve backlog but can raise host load. |
|
||
| Sentry taskscheduler | 0.5 CPU / 512 MiB | ~13% CPU / 387 MiB (76%) | Memory is tight; alert at 85% before it stalls. |
|
||
| Sentry redis | 0.5 CPU / 512 MiB | ~15-30% CPU / 19 MiB | Live and compose cap are aligned. |
|
||
| Sentry uptime-checker | 0.5 CPU / 512 MiB | ~26-30% CPU / 43-187 MiB | Capped after it showed sustained background CPU. |
|
||
| Gitea | 3 CPU / 3 GiB | ~4% CPU / 2.18 GiB (73%) | Good cap; memory headroom is not huge. |
|
||
| GitHub/Gitea runners | unlimited systemd services | one runner had WatchdogSec=5min and 8,490 restarts; `act` CI containers caused load spikes | Must be monitored outside Docker. Remove bad watchdog drop-in and apply per-runner CPU/Memory quotas. |
|
||
| node-exporter | 1 CPU / 256 MiB | ~0-5% CPU / 8 MiB | Good after disabling expensive `arp`, `netclass`, and `netdev` collectors. |
|
||
| cadvisor | 1 CPU / 512 MiB | ~0% CPU / 27 MiB | Good safety cap. |
|
||
| Harbor / Langfuse / Prometheus / Grafana / Nginx | mostly unlimited | currently low | Needs staged limits, but not during Sentry backlog drain. |
|
||
|
||
### 188
|
||
|
||
| Service | Live Limit | Live Usage Snapshot | Verdict |
|
||
|---|---:|---:|---|
|
||
| momo-scheduler | 2 CPU / 2 GiB | ~0.3% CPU / 163 MiB after crawler burst | CPU cap is working. Next fix is crawler concurrency and failed background jobs, not lower CPU. |
|
||
| SignOz ClickHouse | 4 CPU / 24 GiB | ~93-133% CPU / 1.1 GiB | Healthy enough; keep current cap. |
|
||
| SignOz Zookeeper | 1 CPU / 2 GiB | ~8-18% CPU / 1.09 GiB | OK. |
|
||
| cadvisor | 1.5 CPU / 1 GiB | ~0% CPU / 28 MiB | Good. |
|
||
| litellm | 1 CPU / 1 GiB | ~0.5-0.9% CPU / 780 MiB | Good cap; keep stateless mode and do not re-add `DATABASE_URL`. |
|
||
| momo-pro-system | 2 CPU / 2 GiB | ~1-2% CPU / 740 MiB | Good cap; startup cache prewarm must stay single-flight. |
|
||
| momo-scheduler | 2 CPU / 2 GiB | ~0.3% CPU / 105-163 MiB after crawler burst | CPU cap is working. Next fix is crawler concurrency and failed background jobs, not lower CPU. |
|
||
| momo-telegram-bot | 0.5 CPU / 512 MiB | ~0.7% CPU / 66 MiB | Good cap. |
|
||
| momo-db | 2 CPU / 4 GiB | DB had short CPU bursts, then ~0.6-29% with no active long query | Good cap; current bursts are query/workload, not limit pressure. |
|
||
| Monitoring tools / websites / exporters | mostly unlimited | low | Add caps gradually with textfile alerts watching pressure. |
|
||
|
||
## Baseline Policy
|
||
|
||
Use these thresholds for alerting and AI triage:
|
||
|
||
| Signal | Threshold | Action |
|
||
|---|---:|---|
|
||
| host load5 / core | > 1.5 for 15m | Critical SSH diagnosis; classify top service before repair. |
|
||
| Docker container CPU | > 2 cores for 10m | Warning diagnosis; check limit, backlog, and workload type. |
|
||
| Docker container CPU | > 4 cores for 15m | Critical diagnosis; never generic restart. |
|
||
| Docker memory / limit | > 85% for 10m | Warning; raise memory or reduce workload, never lower the limit. |
|
||
| Docker CPU or memory limit | missing for 30m | Warning; add service-specific caps before the next spike. |
|
||
| Docker restarts | > 5 in 15m | Critical; pull logs and fix crash signature. |
|
||
| Systemd runner restarts | > 2 in 15m | Critical; inspect watchdog/drop-ins and active CI jobs. |
|
||
| Systemd runner WatchdogSec | > 0 for 10m | Warning; GitHub Actions runner should not be killed by systemd watchdog. |
|
||
| Systemd runner quota | CPU or memory unlimited for 30m | Warning; apply CPUQuota/MemoryMax or move CI away from Sentry host. |
|
||
| Gitea Actions job runtime | > 20m for 5m | Warning; inspect logs and run `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh` dry-run before stopping stale job containers. The stale-job detector must include both legacy `GITEA-ACTIONS-*` containers and repo-scoped names such as `awoooi-cd-<run>-<job>` / `awoooi-code-review-<run>-<job>`。textfile exporter 也必須輸出 host-side `act_runner` / BuildKit / `next build` process load，因為活躍 CD build 可能不會出現在具名 action container 裡。 |
|
||
|
||
## Rules
|
||
|
||
1. Do not reduce ClickHouse CPU/memory while merges or Kafka backlog are draining.
|
||
2. Do not set ClickHouse `background_pool_size` below the three MergeTree thresholds unless all thresholds are explicitly lower than `pool * ratio`.
|
||
3. Do not use lower memory limits as a load-shedding tool. That creates OOM/restart loops.
|
||
4. For Chrome crawlers, cap concurrency first; CPU caps are only the safety rail.
|
||
5. For Kafka/Snuba, treat high CPU as backlog digestion unless lag stops decreasing.
|
||
6. For monitoring tools, caps are required, but every cap must be paired with self-monitoring.
|
||
7. Every Docker Compose host must emit `docker_container_cpu_cores`, `docker_container_memory_*`, and a restart counter via node-exporter textfile.
|
||
8. Important Docker services must have both CPU and memory guardrails; missing limit alerts should drive service-specific compose fixes, not generic restarts.
|
||
9. Disable node-exporter collectors that are slow or failing on each host; exporter scrape time is part of the resource baseline.
|
||
10. Runner services are part of the resource baseline even when jobs run outside Docker. Export `systemd_unit_*` metrics for all `actions.runner.*` services.
|
||
11. CI runner limits must be set at the systemd unit level. Docker container limits do not protect host load from runner listener, worker, and `act` helper processes.
|
||
12. 當 `docker ps` 沒有 action container 時，優先用 `awoooi_host_gitea_actions_active_process_*` 判讀 110 高 CPU；BuildKit / `runc` / `next build` 可能仍在跑，即使 `awoooi_host_gitea_actions_active_container_count=0`。
|
||
13. Workflow-only commits must not trigger full CD image builds; CD should run only when runtime inputs change.
|
||
14. Stale Gitea Actions cleanup must be dry-run first. Use `--apply` only when the container is older than its workflow/job policy threshold and has no recent useful logs.
|
||
|
||
## Next Safe Rollout Order
|
||
|
||
1. Deploy `scripts/ops/docker-stats-textfile-exporter.py` to 110 and 188 textfile collector cron.
|
||
2. Reload Prometheus rules with the new Docker CPU/memory/restart baseline alerts.
|
||
3. Persist live limits in the owning compose files before considering the host repaired; live `docker update` alone is not durable.
|
||
4. Observe 110 for one drain window after node-exporter collector trim and taskworker concurrency 2. Kafka lag is now near zero; if ClickHouse remains high, tune merge/query behavior or reduce high-volume Sentry features, not Kafka memory.
|
||
5. Tune `momo-scheduler` crawler concurrency on 188; keep 2 CPU / 2 GiB until success rate and latency prove it is too low.
|
||
6. Fix 188 Elephant Alpha/OpenClaw allowed-action drift before enabling resource auto-repair beyond diagnosis.
|
||
7. Add modest caps to currently unlimited low-risk services in small batches. Do not alert every unlimited auxiliary container at once; promote candidates only after 24h usage data.
|
||
8. Deploy `scripts/ops/stop-stale-gitea-actions-jobs.sh` to 110 as `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh`; keep Prometheus auto action in dry-run mode. As of 2026-06-27, it must recognize both legacy `GITEA-ACTIONS-*` and current repo-scoped `*-cd-<run>-*` / `*-code-review-<run>-*` container names；同時 `host-runaway-process-exporter.py` 必須在沒有 action container 可見時仍輸出 BuildKit / `next build` process load。
|
||
9. Fix 110 runner services with sudo-capable host maintenance:
|
||
|
||
```bash
|
||
sudo /home/wooo/scripts/apply-runner-systemd-guardrails.sh --apply
|
||
```
|
||
|
||
## Known Anti-Patterns
|
||
|
||
- `docker restart` as a response to sustained CPU.
|
||
- Lowering ClickHouse merge pool without validating ClickHouse 25.x thresholds.
|
||
- Capping Kafka below current working set during backlog recovery.
|
||
- Treating "no alert" as healthy when cAdvisor or textfile exporters are missing.
|
||
- Letting monitoring collectors spend seconds per scrape; this turns observability into load.
|
||
- Leaving self-hosted runners unlimited on the same host as Sentry/ClickHouse/Gitea.
|
||
- Applying live `docker update` without persisting the same guardrail in compose/systemd/IaC.
|