awoooi/docs/runbooks/HOST-RESOURCE-BASELINE-110-188.md

110/188 Host Resource Baseline

2026-05-05 ogt + Codex Scope: 110 DevOps/Sentry host and 188 AI/Web host Docker Compose resource limits.

Current Live Finding

110

Service	Live Limit	Live Usage Snapshot	Verdict
Sentry ClickHouse	2 CPU / 8 GiB, merge pool 4	capped near 2 cores after pool 8 -> 4 restart	Do not lower memory. CPU quota intentionally slows background merge so Sentry cannot dominate 110. If backlog grows, inspect MergeMutate and Sentry high-volume features before raising it.
Sentry Kafka	2 CPU / 3 GiB	~40-55% CPU / 2.5 GiB (84%)	Memory is close to pressure. Do not reduce memory.
Sentry taskworker	2 CPU / 2 GiB, concurrency 2	~120-181% CPU after restart	Concurrency reduced from 4 to 2 after Kafka lag cleared. Watch Sentry task latency before further changes.
Sentry taskbroker	1 CPU / 512 MiB	~70-98% CPU / 160 MiB	CPU is tight; increasing may improve backlog but can raise host load.
Sentry taskscheduler	0.5 CPU / 512 MiB	~13% CPU / 387 MiB (76%)	Memory is tight; alert at 85% before it stalls.
Sentry redis	0.5 CPU / 512 MiB	~15-30% CPU / 19 MiB	Live and compose cap are aligned.
Sentry uptime-checker	0.5 CPU / 512 MiB	~26-30% CPU / 43-187 MiB	Capped after it showed sustained background CPU.
Gitea	3 CPU / 3 GiB	~4% CPU / 2.18 GiB (73%)	Good cap; memory headroom is not huge.
GitHub/Gitea runners	unlimited systemd services	one runner had WatchdogSec=5min and 8,490 restarts; act CI containers caused load spikes	Must be monitored outside Docker. Remove bad watchdog drop-in and apply per-runner CPU/Memory quotas.
node-exporter	1 CPU / 256 MiB	~0-5% CPU / 8 MiB	Good after disabling expensive arp, netclass, and netdev collectors.
cadvisor	1 CPU / 512 MiB	~0% CPU / 27 MiB	Good safety cap.
Harbor / Langfuse / Prometheus / Grafana / Nginx	mostly unlimited	currently low	Needs staged limits, but not during Sentry backlog drain.

188

Service	Live Limit	Live Usage Snapshot	Verdict
momo-scheduler	2 CPU / 2 GiB	~0.3% CPU / 163 MiB after crawler burst	CPU cap is working. Next fix is crawler concurrency and failed background jobs, not lower CPU.
SignOz ClickHouse	4 CPU / 24 GiB	~93-133% CPU / 1.1 GiB	Healthy enough; keep current cap.
SignOz Zookeeper	1 CPU / 2 GiB	~8-18% CPU / 1.09 GiB	OK.
cadvisor	1.5 CPU / 1 GiB	~0% CPU / 28 MiB	Good.
litellm	1 CPU / 1 GiB	~0.5-0.9% CPU / 780 MiB	Good cap; keep stateless mode and do not re-add DATABASE_URL.
momo-pro-system	2 CPU / 2 GiB	~1-2% CPU / 740 MiB	Good cap; startup cache prewarm must stay single-flight.
momo-scheduler	2 CPU / 2 GiB	~0.3% CPU / 105-163 MiB after crawler burst	CPU cap is working. Next fix is crawler concurrency and failed background jobs, not lower CPU.
momo-telegram-bot	0.5 CPU / 512 MiB	~0.7% CPU / 66 MiB	Good cap.
momo-db	2 CPU / 4 GiB	DB had short CPU bursts, then ~0.6-29% with no active long query	Good cap; current bursts are query/workload, not limit pressure.
Monitoring tools / websites / exporters	mostly unlimited	low	Add caps gradually with textfile alerts watching pressure.

Baseline Policy

Use these thresholds for alerting and AI triage:

Signal	Threshold	Action
host load5 / core	> 1.5 for 15m	Critical SSH diagnosis; classify top service before repair.
Docker container CPU	> 2 cores for 10m	Warning diagnosis; check limit, backlog, and workload type.
Docker container CPU	> 4 cores for 15m	Critical diagnosis; never generic restart.
Docker memory / limit	> 85% for 10m	Warning; raise memory or reduce workload, never lower the limit.
Docker CPU or memory limit	missing for 30m	Warning; add service-specific caps before the next spike.
Docker restarts	> 5 in 15m	Critical; pull logs and fix crash signature.
Systemd runner restarts	> 2 in 15m	Critical; inspect watchdog/drop-ins and active CI jobs.
Systemd runner WatchdogSec	> 0 for 10m	Warning; GitHub Actions runner should not be killed by systemd watchdog.
Systemd runner quota	CPU or memory unlimited for 30m	Warning; apply CPUQuota/MemoryMax or move CI away from Sentry host.
Gitea Actions job runtime	> 20m for 5m	Warning; inspect logs and run /home/wooo/scripts/stop-stale-gitea-actions-jobs.sh dry-run before stopping stale job containers. The stale-job detector must include both legacy GITEA-ACTIONS-* containers and repo-scoped names such as awoooi-cd-<run>-<job> / awoooi-code-review-<run>-<job>。textfile exporter 也必須輸出 host-side act_runner / BuildKit / next build process load，因為活躍 CD build 可能不會出現在具名 action container 裡。

Rules

1. Do not reduce ClickHouse CPU/memory while merges or Kafka backlog are draining.
2. Do not set ClickHouse background_pool_size below the three MergeTree thresholds unless all thresholds are explicitly lower than pool * ratio.
3. Do not use lower memory limits as a load-shedding tool. That creates OOM/restart loops.
4. For Chrome crawlers, cap concurrency first; CPU caps are only the safety rail.
5. For Kafka/Snuba, treat high CPU as backlog digestion unless lag stops decreasing.
6. For monitoring tools, caps are required, but every cap must be paired with self-monitoring.
7. Every Docker Compose host must emit docker_container_cpu_cores, docker_container_memory_*, and a restart counter via node-exporter textfile.
8. Important Docker services must have both CPU and memory guardrails; missing limit alerts should drive service-specific compose fixes, not generic restarts.
9. Disable node-exporter collectors that are slow or failing on each host; exporter scrape time is part of the resource baseline.
10. Runner services are part of the resource baseline even when jobs run outside Docker. Export systemd_unit_* metrics for all actions.runner.* services.
11. CI runner limits must be set at the systemd unit level. Docker container limits do not protect host load from runner listener, worker, and act helper processes.
12. 當 docker ps 沒有 action container 時，優先用 awoooi_host_gitea_actions_active_process_* 判讀 110 高 CPU；BuildKit / runc / next build 可能仍在跑，即使 awoooi_host_gitea_actions_active_container_count=0。
13. Workflow-only commits must not trigger full CD image builds; CD should run only when runtime inputs change.
14. Stale Gitea Actions cleanup must be dry-run first. Use --apply only when the container is older than its workflow/job policy threshold and has no recent useful logs.

Next Safe Rollout Order

1. Deploy scripts/ops/docker-stats-textfile-exporter.py to 110 and 188 textfile collector cron.
2. Reload Prometheus rules with the new Docker CPU/memory/restart baseline alerts.
3. Persist live limits in the owning compose files before considering the host repaired; live docker update alone is not durable.
4. Observe 110 for one drain window after node-exporter collector trim and taskworker concurrency 2. Kafka lag is now near zero; if ClickHouse remains high, tune merge/query behavior or reduce high-volume Sentry features, not Kafka memory.
5. Tune momo-scheduler crawler concurrency on 188; keep 2 CPU / 2 GiB until success rate and latency prove it is too low.
6. Fix 188 Elephant Alpha/OpenClaw allowed-action drift before enabling resource auto-repair beyond diagnosis.
7. Add modest caps to currently unlimited low-risk services in small batches. Do not alert every unlimited auxiliary container at once; promote candidates only after 24h usage data.
8. Deploy scripts/ops/stop-stale-gitea-actions-jobs.sh to 110 as /home/wooo/scripts/stop-stale-gitea-actions-jobs.sh; keep Prometheus auto action in dry-run mode. As of 2026-06-27, it must recognize both legacy GITEA-ACTIONS-* and current repo-scoped *-cd-<run>-* / *-code-review-<run>-* container names；同時 host-runaway-process-exporter.py 必須在沒有 action container 可見時仍輸出 BuildKit / next build process load。
9. Fix 110 runner services with sudo-capable host maintenance:

sudo /home/wooo/scripts/apply-runner-systemd-guardrails.sh --apply

Known Anti-Patterns

• docker restart as a response to sustained CPU.
• Lowering ClickHouse merge pool without validating ClickHouse 25.x thresholds.
• Capping Kafka below current working set during backlog recovery.
• Treating "no alert" as healthy when cAdvisor or textfile exporters are missing.
• Letting monitoring collectors spend seconds per scrape; this turns observability into load.
• Leaving self-hosted runners unlimited on the same host as Sentry/ClickHouse/Gitea.
• Applying live docker update without persisting the same guardrail in compose/systemd/IaC.