wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

fix(ops): align stale job cleanup thresholds
All checks were successful
Code Review / ai-code-review (push) Successful in 28s
Details
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Successful in 36s
Details

Browse Source
This commit is contained in:
Your Name
2026-05-05 14:54:17 +08:00
parent 5e625f777d
commit 72d66e4ae6
5 changed files with 36 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
ops/monitoring/alerts-unified.yml
View File

@@ -678,7 +678,7 @@ groups:
summary: "Gitea Actions job {{ $labels.container_name }} 執行超過 20 分鐘"
description: "{{ $labels.container_name }} 已超過 20 分鐘，110 曾出現 code-review/CD job timeout 失效而持續佔用 runner。"
auto_repair_action: "ssh 192.168.0.{{ $labels.host }} 'docker ps --filter name={{ $labels.container_name }}; docker logs --tail=120 {{ $labels.container_name }}; bash /home/wooo/scripts/stop-stale-gitea-actions-jobs.sh --min-age-seconds 1200'"
runbook: "先執行 dry-run；若 logs 空白且超過 workflow timeout，再用 --apply 停止 stale job container。不要停止仍在輸出或正在部署的 job。"
runbook: "先執行 dry-run；清理腳本會依 workflow/job 名稱套停止門檻。若 logs 空白且超過該 job timeout buffer，再用 --apply 停止 stale job container。不要停止仍在輸出或正在部署的 job。"
- alert: SystemdRunnerRestartSpike
# 2026-05-05 ogt + Codex: 110 GitHub Actions runner watchdog loop was outside Docker/cAdvisor coverage.

2
ops/monitoring/alerts.yml
View File

@@ -684,7 +684,7 @@ groups:
summary: "Gitea Actions job {{ $labels.container_name }} 執行超過 20 分鐘"
description: "{{ $labels.container_name }} 已超過 20 分鐘，110 曾出現 code-review/CD job timeout 失效而持續佔用 runner。"
auto_repair_action: "ssh 192.168.0.{{ $labels.host }} 'docker ps --filter name={{ $labels.container_name }}; docker logs --tail=120 {{ $labels.container_name }}; bash /home/wooo/scripts/stop-stale-gitea-actions-jobs.sh --min-age-seconds 1200'"
runbook: "先執行 dry-run；若 logs 空白且超過 workflow timeout，再用 --apply 停止 stale job container。不要停止仍在輸出或正在部署的 job。"
runbook: "先執行 dry-run；清理腳本會依 workflow/job 名稱套停止門檻。若 logs 空白且超過該 job timeout buffer，再用 --apply 停止 stale job container。不要停止仍在輸出或正在部署的 job。"
- alert: SystemdRunnerRestartSpike
# 2026-05-05 ogt + Codex: 110 GitHub Actions runner watchdog loop was outside Docker/cAdvisor coverage.