fix(ops): recognize repo-scoped CI containers in load guard [skip ci]

This commit is contained in:
ogt
2026-06-27 12:45:01 +08:00
parent c4fcd9cb12
commit 7f706feded
6 changed files with 63 additions and 9 deletions

View File

@@ -54,6 +54,33 @@
- 本段只修補前端 message catalog不改 executor、Telegram、主機、K8s、runtime gate、secret 或 live apply 權限。
- P2-415 仍維持 controlled executor dispatch / live apply / Telegram send / Bot API / host write / kubectl / destructive operation 全部 `0`
## 2026-06-2712:43 110 CPU 高負載實際防護CI / smoke container 辨識修正並部署
**時間與來源**
- 2026-06-27 12:40-12:43 Asia/Taipei。
- 來源110 `ps` / Docker stats / host runaway exporter / stale Gitea action dry-run / Gitea log readback。
**實際判讀**
- 12:40 `load average` 已從前一輪高峰下降到 `4.87, 9.98, 40.74`,主要高峰來源已從 stockPlatform smoke 轉為 AWOOOI 正式 CD `awoooi-cd-5901-1-e2e-smoke`、Playwright / Chrome / ffmpeg 與 Gitea log/update該 job 當時僅跑十幾秒,未判為 stale。
- 12:43 `load average` 進一步下降到 `2.28, 6.29, 33.05`active CI container 已歸零,`gitea``55%` container CPU未見 30 分鐘以上 orphan browser group。
- 缺口不是完全沒有監控,而是 `host-runaway-process-exporter.py` 舊版只計算 `GITEA-ACTIONS-*`,漏掉現行 `awoooi-cd-<run>-<job>` / `awoooi-code-review-<run>-<job>` 命名,導致 active action container 會誤報 `0`
**修復內容**
- `scripts/ops/host-runaway-process-exporter.py` 已支援 `GITEA-ACTIONS-*``*-cd-<run>-*``*-code-review-<run>-*` container 命名。
- `scripts/ops/stop-stale-gitea-actions-jobs.sh` 已支援相同命名,並給 deploy / tests / post-deploy / code-review / e2e-smoke / source-link-smoke 不同 stale threshold預設仍 dry-run。
- `scripts/reboot-recovery/awoooi-startup-110.sh` 的 runner drain 判斷也同步支援現行 container 命名,避免下次重啟時漏判正在跑的 CD task。
- 已部署到 110`/home/wooo/scripts/host-runaway-process-exporter.py``/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh``/usr/local/bin/awoooi-startup-110.sh`
**驗證**
- `pytest scripts/ops/tests/test_host_runaway_process_exporter.py -q``7 passed`
- `bash -n scripts/ops/stop-stale-gitea-actions-jobs.sh scripts/reboot-recovery/awoooi-startup-110.sh`:通過。
- 110 exporter readback`awoooi_host_gitea_actions_active_container_count{host="110"} 0``awoooi_host_runaway_browser_orphan_group_count ... 0``awoooi_host_load5_per_core{host="110"} 0.524167`
- 110 stale job dry-run`No stale Gitea Actions containers older than policy threshold`
**邊界**
- 本段沒有重啟 Docker / Gitea / Nginx / systemd service / DB / K3s也沒有 firewall / secret / active response。
- 這是實際主機防護工具更新與部署,不是放寬 runtime gate自動修復仍需 owner / maintenance window / evidence gates。
## 2026-06-27P2-415 AI Agent 受控 Executor 交接跑道API / 前台 / 測試完成
**背景**P2-409 已把 high 風險從「人工 owner review」調整為可走 controlled apply queuecritical 仍保留 break-glass 邊界;本段承接 P2-409 / P2-410 / P2-411補上可被產品與正式 API 讀回的受控 executor handoff 跑道,避免只停留在 UI 文案或口頭批准。

View File

@@ -52,7 +52,7 @@ Use these thresholds for alerting and AI triage:
| Systemd runner restarts | > 2 in 15m | Critical; inspect watchdog/drop-ins and active CI jobs. |
| Systemd runner WatchdogSec | > 0 for 10m | Warning; GitHub Actions runner should not be killed by systemd watchdog. |
| Systemd runner quota | CPU or memory unlimited for 30m | Warning; apply CPUQuota/MemoryMax or move CI away from Sentry host. |
| Gitea Actions job runtime | > 20m for 5m | Warning; inspect logs and run `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh` dry-run before stopping stale job containers. |
| Gitea Actions job runtime | > 20m for 5m | Warning; inspect logs and run `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh` dry-run before stopping stale job containers. The stale-job detector must include both legacy `GITEA-ACTIONS-*` containers and repo-scoped names such as `awoooi-cd-<run>-<job>` / `awoooi-code-review-<run>-<job>`. |
## Rules
@@ -79,7 +79,7 @@ Use these thresholds for alerting and AI triage:
5. Tune `momo-scheduler` crawler concurrency on 188; keep 2 CPU / 2 GiB until success rate and latency prove it is too low.
6. Fix 188 Elephant Alpha/OpenClaw allowed-action drift before enabling resource auto-repair beyond diagnosis.
7. Add modest caps to currently unlimited low-risk services in small batches. Do not alert every unlimited auxiliary container at once; promote candidates only after 24h usage data.
8. Deploy `scripts/ops/stop-stale-gitea-actions-jobs.sh` to 110 as `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh`; keep Prometheus auto action in dry-run mode.
8. Deploy `scripts/ops/stop-stale-gitea-actions-jobs.sh` to 110 as `/home/wooo/scripts/stop-stale-gitea-actions-jobs.sh`; keep Prometheus auto action in dry-run mode. As of 2026-06-27, it must recognize both legacy `GITEA-ACTIONS-*` and current repo-scoped `*-cd-<run>-*` / `*-code-review-<run>-*` container names.
9. Fix 110 runner services with sudo-capable host maintenance:
```bash