From da8456cf071e01a43299aa655334d6d2baae58ec Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 21 May 2026 19:10:21 +0800 Subject: [PATCH] docs(awooop): record t135 runner ownership evidence [skip ci] --- docs/LOGBOOK.md | 104 ++++++++++++++++++ ...-04-15-MASTER-ai-autonomous-flywheel-v2.md | 8 ++ 2 files changed, 112 insertions(+) diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 8dcb6d29d..5842bc4d5 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,107 @@ +## 2026-05-21｜T135 Gitea runner ownership drain 與 stale network cleanup + +**觸發**： + +- T134 已清掉 runner workspace cleanup noise，但 live 110 仍同時跑兩個 act-runner： + - host-level `gitea-act-runner-host.service` + - Docker-wrapped `gitea-runner` +- Docker-wrapped runner 的 restart policy 是 `unless-stopped`，且 live log 顯示最近 2 小時實際接過 `awoooi`、`stockplatform-v2`、`ewoooc` tasks。 +- 這與 `ops/runner/README.md` 既有結論衝突：110 應只保留 host-level runner，Docker-wrapped `gitea-runner` 必須停用，否則會搶 job、污染 CD ownership 與 build pressure 判讀。 + +**Live 收斂**： + +- 檢查 AWOOI 最新 runs：無 active AWOOI run。 +- 檢查其他 repo：`stockplatform-v2` / `ewoooc` 當時有 active task，因此沒有直接硬停 Docker runner。 +- 對 live `gitea-runner` 執行 drain： + - `docker update --restart=no gitea-runner` + - `docker kill --signal=SIGINT gitea-runner` +- Docker runner log 明確出現： + - `runner: wooo-runner shutdown initiated, waiting 1h0m0s for running jobs to complete before shutting down` +- 後續 readback： + - `Restart=no Status=exited Running=false` + - 沒有 `GITEA-ACTIONS-TASK-*` 殘留。 + - 空 `b5-test-net` 已移除。 + - 110 live `docker-health-monitor.sh` 仍排除 `gitea-runner`，不會把它拉回來。 + +**Repo 修正**： + +- `ops/runner/install-gitea-host-runner-service.sh` + - 新增 `gitea_task_containers_running()`。 + - 停 legacy Docker runner 前先關 restart policy。 + - 若有 active task container，改送 `SIGINT` drain，而不是直接 `docker stop`。 + - 若無 active task，用長 timeout stop，避免 10 秒預設 timeout 誤殺。 +- `scripts/reboot-recovery/awoooi-startup-110.sh` + - 同步改成 active task 時 drain Docker runner。 +- `scripts/ci/cleanup-host-runner-workspace.sh` + - cleanup 時若 `b5-test-net` 已無 containers，移除 stale network。 +- `ops/runner/README.md` + - 新增第五層修復：legacy Docker runner drain。 + +**Verification / deploy**： + +```text +Local: +bash -n ops/runner/install-gitea-host-runner-service.sh scripts/reboot-recovery/awoooi-startup-110.sh scripts/ci/cleanup-host-runner-workspace.sh -> pass +git diff --check -> pass + +Code commit: +9b465ee1 ci(runner): drain legacy docker act runner safely + +Gitea Actions: +#2815 ai-code-review -> success +#2816 workflow_dispatch CD -> success + tests job 3634 -> success + pytest: 2190 passed, 23 skipped + B5 integration: 5 passed + cleanup: host runner workspace artifacts cleaned + build-and-deploy job 3635 -> success + duration: 10:58:35Z -> 11:03:50Z + post-deploy-checks job 3636 -> success + Alert Chain Smoke: 8/8 checks passed + E2E smoke: 5 passed + CI/CD success notification mirrored through AWOOI API + cleanup: host runner workspace artifacts cleaned + +Deploy marker: +5aa46bc9 chore(cd): deploy 9b465ee [skip ci] + +Production readback: +GET https://awoooi.wooo.work/api/v1/health + -> healthy, prod, mock_mode=false + -> api/postgresql/redis/ollama/openclaw/signoz up +K8s: + awoooi-api 192.168.0.110:5000/awoooi/api:9b465ee140a32ac73742d59f589164cbce798d04 2/2 + awoooi-web 192.168.0.110:5000/awoooi/web:9b465ee140a32ac73742d59f589164cbce798d04 2/2 + awoooi-worker 192.168.0.110:5000/awoooi/api:9b465ee140a32ac73742d59f589164cbce798d04 1/1 +110 runner: + gitea-runner -> Restart=no Status=exited Running=false + b5-test-net -> absent + load average after deploy -> about 3.72, 3.80, 10.09 +``` + +**判讀**： + +- T135 將 runner ownership 從「雙 runner 搶工」收斂到「host-level runner 單一主控」。 +- 這不是只改文件：live Docker runner 已安全 drain 並退出；CD #2816 已在此狀態下完成。 +- Build pressure 顯著改善，本輪 `build-and-deploy` 約 5 分 15 秒完成；T134 的 `#2803` 同段約 20 分鐘。 +- 後續 tech debt： + - 仍應做 Dockerfile build pressure cleanup：API image `apt-get` / `chown -R appuser` 分層、Web `next build` cache / offload。 + - 若其他 repo 仍大量推送，host runner capacity=1 會讓隊列變長；下一階段要做 runner pool / repo label 隔離，而不是重新啟用 Docker-wrapped runner。 + - Playwright apt duplicate source warnings 仍屬 CI image/source-list hygiene。 + +**目前整體進度**： + +- AwoooP 告警可觀測鏈：99.998%。 +- Incident-level source correlation 可見性：98.8%。 +- Source correlation apply 狀態鏈可驗證性：99.72%。 +- Source correlation freshness / rolling gate：98.2%。 +- 前端 AI 自動化管理介面同步：99.999%。 +- Dashboard snapshot / SSE console noise 收斂：99.2%。 +- CI/CD runner hygiene：約 98.8% → 99.2%。 +- Runner ownership 收斂：約 72% → 96%。 +- Build host pressure治理：約 76% → 82%。 +- 完整 AI 自動化管理產品化：99.959% → 99.960%。 + ## 2026-05-21｜T134 Gitea runner workspace cleanup 與 110 hygiene pass **觸發**： diff --git a/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md b/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md index 474dc0f13..fc61e8e46 100644 --- a/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md +++ b/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md @@ -2657,6 +2657,14 @@ Phase 6 完成後 - 判讀：T134 收斂 runner workspace cleanup false noise，讓 CI/CD log 更接近真訊號；不等於 runner ownership 完全治理。下一段應處理 110 host-level act_runner 與 Docker-wrapped `gitea-runner` 雙 daemon 衝突、stale `b5-test-net`、API image `apt-get` / `chown -R appuser` 與 Web `next build` 的 20 分鐘 build pressure、Playwright apt duplicate source warnings。 - 目前進度更新：AwoooP 告警可觀測鏈約 99.998%；Incident-level source correlation 可見性約 98.8%；Source correlation apply 狀態鏈可驗證性約 99.72%；Source correlation freshness / rolling gate 約 98.2%；前端 AI 自動化管理介面同步約 99.999%；Dashboard snapshot / SSE console noise 收斂約 99.2%；CI/CD runner hygiene 約 98.8%；Build host pressure治理約 76%；完整 AI 自動化管理產品化約 99.959%。 +**T135 Gitea runner ownership drain 與 stale network cleanup（2026-05-21 台北）**： +- 觸發：T134 後 live 110 仍同時跑 host-level `gitea-act-runner-host.service` 與 Docker-wrapped `gitea-runner`；Docker runner restart policy 是 `unless-stopped`，且最近 2 小時實際接過 `awoooi` / `stockplatform-v2` / `ewoooc` tasks，與「只保留 host runner」的 runner ownership 目標衝突。 +- Live 收斂：未直接硬停 active job；先確認 AWOOI 無 active run，再對 Docker runner 執行 `docker update --restart=no gitea-runner` 與 `docker kill --signal=SIGINT gitea-runner`。Live log 顯示 `shutdown initiated, waiting 1h0m0s...`；readback 為 `Restart=no Status=exited Running=false`。空 `b5-test-net` 已移除，live docker-health-monitor 仍排除 `gitea-runner`。 +- Repo 修正：`ops/runner/install-gitea-host-runner-service.sh` 與 `scripts/reboot-recovery/awoooi-startup-110.sh` 改為 active task 時送 SIGINT drain，不用預設 10 秒 `docker stop`；`scripts/ci/cleanup-host-runner-workspace.sh` 會移除空 `b5-test-net`；`ops/runner/README.md` 補第五層 legacy Docker runner drain。 +- Verification：`bash -n`、`git diff --check` pass。`9b465ee1 ci(runner): drain legacy docker act runner safely` 已推 Gitea main；Code Review #2815 success；手動 CD #2816 success：tests job 3634 `2190 passed, 23 skipped` + B5 `5 passed` + cleanup clean，build-and-deploy job 3635 success（約 5m15s），post-deploy job 3636 Alert Chain `8/8` + E2E `5 passed` + AWOOI API success notification mirrored + cleanup clean。Deploy marker `5aa46bc9 chore(cd): deploy 9b465ee [skip ci]`；production health healthy/prod/mock_mode=false；API/Web image `9b465ee140a32ac73742d59f589164cbce798d04` 2/2，Worker 1/1 ready。110 runner readback：Docker `gitea-runner` exited/restart=no，`b5-test-net` absent，load 約 3.72/3.80/10.09。 +- 判讀：T135 已把 runner ownership 從雙 runner 搶工收斂到 host runner 單一主控；下一段不要重新啟用 Docker-wrapped runner，而是做 runner pool / repo label 隔離、API image `apt-get` / `chown -R` 分層、Web build cache/offload、Playwright apt source-list hygiene。 +- 目前進度更新：AwoooP 告警可觀測鏈約 99.998%；Incident-level source correlation 可見性約 98.8%；Source correlation apply 狀態鏈可驗證性約 99.72%；Source correlation freshness / rolling gate 約 98.2%；前端 AI 自動化管理介面同步約 99.999%；Dashboard snapshot / SSE console noise 收斂約 99.2%；CI/CD runner hygiene 約 99.2%；Runner ownership 收斂約 96%；Build host pressure治理約 82%；完整 AI 自動化管理產品化約 99.960%。 + --- ### 2026-04-20 晚 (台北) — C1-C4 全流程串接 — Playbook 鏈路保護（commit de2d34d）