fix(ops): alert on stale gitea actions jobs
Some checks failed
CD Pipeline / tests (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled
CD Pipeline / post-deploy-checks (push) Has been cancelled
Code Review / ai-code-review (push) Has been cancelled
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Has been cancelled
Some checks failed
CD Pipeline / tests (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled
CD Pipeline / post-deploy-checks (push) Has been cancelled
Code Review / ai-code-review (push) Has been cancelled
Deploy Alert Rules / Deploy Prometheus Alert Rules (push) Has been cancelled
This commit is contained in:
@@ -1605,6 +1605,8 @@ psql $DATABASE_URL -f apps/api/migrations/cleanup_duplicate_deprecated_playbooks
|
||||
| `0e14935` | `SystemdRunner*` early classification + `{unit}` template variable |
|
||||
| `ab0f0a8` | deploy API image `runner-classify-20260505-0e14935` |
|
||||
| `2e128f9` | Gitea Code Review stale-run guard,避免快速連推堆疊多個 runner job |
|
||||
| `3b73cc7` | CD paths 收斂,workflow-only commits 不再觸發完整 image build/deploy |
|
||||
| `pending` | Docker textfile 補 `docker_container_started_seconds` + `DockerGiteaActionsJobStale` |
|
||||
|
||||
### 下一步
|
||||
|
||||
|
||||
@@ -46,6 +46,7 @@ Use these thresholds for alerting and AI triage:
|
||||
| Systemd runner restarts | > 2 in 15m | Critical; inspect watchdog/drop-ins and active CI jobs. |
|
||||
| Systemd runner WatchdogSec | > 0 for 10m | Warning; GitHub Actions runner should not be killed by systemd watchdog. |
|
||||
| Systemd runner quota | CPU or memory unlimited for 30m | Warning; apply CPUQuota/MemoryMax or move CI away from Sentry host. |
|
||||
| Gitea Actions job runtime | > 20m for 5m | Warning; inspect logs and stop stale job containers if they outlive workflow timeout. |
|
||||
|
||||
## Rules
|
||||
|
||||
@@ -59,6 +60,7 @@ Use these thresholds for alerting and AI triage:
|
||||
8. Disable node-exporter collectors that are slow or failing on each host; exporter scrape time is part of the resource baseline.
|
||||
9. Runner services are part of the resource baseline even when jobs run outside Docker. Export `systemd_unit_*` metrics for all `actions.runner.*` services.
|
||||
10. CI runner limits must be set at the systemd unit level. Docker container limits do not protect host load from runner listener, worker, and `act` helper processes.
|
||||
11. Workflow-only commits must not trigger full CD image builds; CD should run only when runtime inputs change.
|
||||
|
||||
## Next Safe Rollout Order
|
||||
|
||||
|
||||
Reference in New Issue
Block a user