From 899635cc63a257c7298ac1852d4e0f28cfd69f55 Mon Sep 17 00:00:00 2001 From: Your Name Date: Sun, 28 Jun 2026 08:44:45 +0800 Subject: [PATCH] docs(runner): record 110 fail-closed pressure exception [skip ci] --- AGENTS.md | 2 ++ docs/HARD_RULES.md | 14 ++++++++++++++ docs/LOGBOOK.md | 17 +++++++++++++++++ ...26-04-15-MASTER-ai-autonomous-flywheel-v2.md | 2 +- ops/runner/README.md | 13 +++++++++++++ scripts/ci/wait-host-web-build-pressure.sh | 3 ++- 6 files changed, 49 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b5377a5c8..750ee049e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,6 +46,8 @@ 正確動作是 AI 自動補齊 target selector、source-of-truth diff、check-mode / dry-run、rollback、post-apply verifier、KM / PlayBook trust writeback，然後推進可驗證、可回滾、低爆炸半徑的實作。 +**110 runner 壓力事故例外**：Gitea / act-runner 對 110 造成 CPU / headless smoke 壓力時，屬事故級容量保護，不得用「全面授權」直接重開 runner、移除 mask、還原 runner binary 或把 host pressure gate 改成 warn-only。正確動作是先做 runner 搬遷 / 限流 / label isolation / smoke 排程，再以 check-mode、rollback 與 post-apply verifier 受控恢復。 + --- ## 🔴 絕對禁止 → [HARD_RULES.md](docs/HARD_RULES.md) diff --git a/docs/HARD_RULES.md b/docs/HARD_RULES.md index 88e3ef243..39580d3c1 100644 --- a/docs/HARD_RULES.md +++ b/docs/HARD_RULES.md @@ -287,6 +287,20 @@ OpenClaw 核心替換、仲裁模型升級、SDK / runtime 新依賴正式引入 force push / 刪 repo / 刪 refs / 改 repo visibility / raw runtime secret volume 讀寫 ``` +### 110 runner 壓力事故例外 + +2026-06-28 事故後，110 上的 Gitea / act-runner、StockPlatform headless smoke、host-side Next build 與 Docker / BuildKit 壓力屬容量事故保護面。即使收到「批准 / 繼續 / 全面授權」，也不得直接重開 runner、解除 service mask、還原 live runner binary、恢復泛用 `ubuntu-latest` label，或把 host pressure gate 改成 warn-only 作為預設。 + +允許的 controlled apply 是降壓與防再發：停止 / disable / mask runner、quarantine runner binary、收斂 labels、補 source fail-closed guard、搬遷 runner、限制 concurrency、把 smoke 改成排程 / 非 110 runner，以及執行只讀 pressure / cold-start verifier。 + +恢復 runner 必須同時具備： + +1. target selector：明確列出 service、runner dir、label 與承接 repo。 +2. source-of-truth diff：repo / unit / startup script / runner config 都有一致變更。 +3. 限流或搬遷：不再由 110 production host 承接泛用重型 build / smoke。 +4. rollback：能回到 inactive / masked / fail-closed stub。 +5. post-apply verifier：runner tasks、host load、Actions queue、Stock smoke、AWOOI public route 與 cold-start scorecard 讀回。 + ### 跨專案繼承 本節是 AWOOOI / AwoooP / IwoooS / StockPlatform / VibeWork / MOMO / AwoooGo / 2026FIFA / Agent Bounty / Tsenyang 等所有專案的預設行為。若任一 repo 的 `AGENTS.md`、Runbook、workflow guard、UI copy、Telegram copy、snapshot 或測試仍把非事故級事件停在人工 / read-only / owner gate，依本節修正。 diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 2a2cc90a2..975a51ea6 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,20 @@ +## 2026-06-28 — 08:45 110 runner 壓力事故 source / live fail-closed 收斂 + +**背景**：統帥全面授權打開非事故級 gate，但 110 Gitea runner 反覆拉起 StockPlatform headless Chrome smoke，已造成 production host CPU / CI 壓力事故；runner 未搬遷 / 限流前不得直接重開。 + +**完成內容**： +- Gitea main / branch 已推 `54eea881e fix(ci): fail closed host runner pressure guards [skip ci]`，回復 `scripts/ci/wait-host-web-build-pressure.sh` fail-closed 預設、`scripts/reboot-recovery/awoooi-startup-110.sh` 雙鑰匙 sentinel、`.gitea/workflows/cd.yaml` Docker build lock fail-closed。 +- Live 110 `/usr/local/bin/awoooi-wait-host-web-build-pressure.sh` 與 `/usr/local/bin/awoooi-startup-110.sh` 已同步 fail-closed 並加 immutable。 +- Live 110 四條 runner 入口改為 immutable fail-closed stub，原 ELF 僅 quarantine 不讀內容：`/home/wooo/act-runner/act_runner`、`/home/wooo/act-runner/act_runner.real-20260628-runner-pressure-guard`、`/home/wooo/act-runner-controlled/act_runner`、`/home/wooo/awoooi-controlled-runner/awoooi_controlled_runner`。 +- `gitea-act-runner-host.service`、`gitea-act-runner-awoooi-controlled.service`、`gitea-awoooi-controlled-runner.service`、`gitea-act-runner-awoooi-open.service` 讀回 inactive / disabled，並加上 mask 防止直啟。 +- `AGENTS.md`、`docs/HARD_RULES.md`、`ops/runner/README.md` 與 MASTER runner 章節補上 110 runner 壓力事故例外：全面授權不等於可重開 production host runner。 + +**Live readback**： +- 08:41 讀回四個 runner binary 皆為 163-byte shell stub、immutable、執行時回 fail-closed 訊息；runner / StockPlatform / Playwright / Chrome tasks 為 `0`。 +- 08:40 曾抓到 parent=1 的孤兒 `/home/wooo/act-runner/act_runner.real-20260628-runner-pressure-guard daemon --config config.yaml`，已 terminate；後續改為 stub 以防任何 direct exec 繞過 systemd。 + +**邊界**：本段沒有重啟 Docker / Nginx / firewall / K3s / DB，沒有讀 raw sessions / SQLite / auth / `.env` / runner token。仍未宣稱全主機 scorecard green；下一步是 runner 搬遷 / 限流、StockPlatform smoke 排程或移出 110、再跑 cold-start scorecard。 + ## 2026-06-28 — 03:28 Wazuh manager registry acceptance 收件驗證 production 收斂 **完成內容**： diff --git a/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md b/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md index 6420d8d45..0abeff3ff 100644 --- a/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md +++ b/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md @@ -3865,7 +3865,7 @@ Phase 6 完成後 **T132 CD host Web build pressure gate（2026-05-21 台北）**： - 觸發：T131 production deploy 期間再次觀察到 110 runner 同時跑 AWOOI Web image build 與 `stockplatform-v2` host-side Next build；既有 workflow concurrency 與 Docker network lock 不會阻擋其他 repo 的 host-side `next build` / `turbo build`。 -- 修正：`.gitea/workflows/cd.yaml` 在 Harbor login 後、Docker build lock 前新增 `Wait for Host Web Build Pressure`；新增 `scripts/ci/wait-host-web-build-pressure.sh` 讀取 `ps` 偵測外部 frontend production build，排除 AWOOI checkout / local worktree / `/app/apps/web` 自身 Web Docker build。預設等待 60 次、每次 10 秒，逾時先 warning 放行；`HOST_WEB_BUILD_PRESSURE_WARN_ONLY=0` 可在 runner 隔離更穩後切 hard fail。`ops/runner/README.md` 補第四層 runner 修復與長期 runner isolation / build offload 方向。 +- 修正：`.gitea/workflows/cd.yaml` 在 Harbor login 後、Docker build lock 前新增 `Wait for Host Web Build Pressure`；新增 `scripts/ci/wait-host-web-build-pressure.sh` 讀取 `ps` 偵測外部 frontend production build，排除 AWOOI checkout / local worktree / `/app/apps/web` 自身 Web Docker build。2026-06-28 事故後已改為預設等待 60 次、每次 10 秒，逾時 hard fail；只有明確設定 `HOST_WEB_BUILD_PRESSURE_WARN_ONLY=1` 的受控補跑才可 warning 放行。`ops/runner/README.md` 補第四層 runner 修復與長期 runner isolation / build offload 方向。 - Verification：workflow YAML parse ok；`bash -n scripts/ci/wait-host-web-build-pressure.sh` pass；`git diff --check` pass；fixture 驗證 `stockplatform-v2 ... next build` 會被抓到，`/workspace/wooo/awoooi ... next build` 與 `/app/apps/web ... next/dist/bin/next build` 會被排除。`b3ab4da0 ci(cd): wait for host web build pressure` 已推 Gitea main；Code Review #2783 completed/success。受控 CD workflow_dispatch #2784 completed/success：tests 3578 / build-and-deploy 3579 / post-deploy-checks 3580 全綠；真實 log 顯示 gate 抓到 `stockplatform-v2 ... next build`、等待 10s 後才取得 Docker build lock；deploy marker `c44188b chore(cd): deploy 251f5ad [skip ci]`；ArgoCD Synced + Healthy；API/Web/Worker rollout success；post-deploy E2E 5 passed；CI/CD success notification mirrored through AWOOI API。Production readback：health healthy/prod/mock_mode=false；API/Web image `251f5ad6580676fb61094ad4fd44124f179ba0d5`，API 2/2、Web 2/2、Worker 1/1 ready。 - 判讀：T132 是 shared runner 尚未拆分前的保守等待門，不等於 runner overload 結構性完結；它降低跨 repo frontend build 疊加導致 AWOOI CD / AwoooP 可觀測誤判的機率。下一層若仍看到 110 pressure，應推 runner isolation / build offload。同步暴露 cleanup 候選：`apps/web/Dockerfile` legacy ENV warning 與 act runner cleanup symlink/permission warning。 - 目前進度更新：AwoooP 告警可觀測鏈約 99.998%；Incident-level source correlation 可見性約 98.8%；Source correlation apply 狀態鏈可驗證性約 99.72%；Source correlation freshness / rolling gate 約 98.2%；前端 AI 自動化管理介面同步約 99.999%；Dashboard snapshot / SSE console noise 收斂約 99.2%；CI/CD runner hygiene 約 97.8%；完整 AI 自動化管理產品化約 99.957%。 diff --git a/ops/runner/README.md b/ops/runner/README.md index bd1d015bf..bc9773f9e 100644 --- a/ops/runner/README.md +++ b/ops/runner/README.md @@ -386,6 +386,19 @@ runner registration / service： 狀態；AWOOI workflows 也應只使用 `awoooi-ubuntu` 或 `awoooi-host`，不可再使用 `ubuntu-latest` / `self-hosted` 這類泛用 label。這不代表 runner 搬遷完成，也不代表可以直接重開 runner。 +2026-06-28 live update：110 runner 壓力事故確認有直呼 +`/home/wooo/act-runner/act_runner.real-20260628-runner-pressure-guard` 的孤兒 +daemon。四條 live runner 入口已改為 immutable fail-closed stub，原 ELF 僅 +quarantine 不讀內容；相關 systemd units 維持 inactive / disabled / masked： + +- `/home/wooo/act-runner/act_runner` +- `/home/wooo/act-runner/act_runner.real-20260628-runner-pressure-guard` +- `/home/wooo/act-runner-controlled/act_runner` +- `/home/wooo/awoooi-controlled-runner/awoooi_controlled_runner` + +未完成 runner 搬遷 / 限流 / smoke 排程前，不得解除 mask、還原 ELF、恢復 +泛用 runner label，或把 host pressure gate 預設改成 warn-only。 + --- 版本: v2.0 | 更新: 2026-03-29 | 作者: Claude Code 變更: v1.0→v2.0 序列建構取代 Job Concurrency Groups diff --git a/scripts/ci/wait-host-web-build-pressure.sh b/scripts/ci/wait-host-web-build-pressure.sh index 29e783841..f4dfd7a7b 100755 --- a/scripts/ci/wait-host-web-build-pressure.sh +++ b/scripts/ci/wait-host-web-build-pressure.sh @@ -7,7 +7,8 @@ set -euo pipefail # production host and a CI host, so CD must not start a new Docker/Next build # while load, BuildKit, Gitea Actions, or headless smoke pressure is already high. # This gate never kills, renices, or rewrites another repo's process tree. -# 2026-06-28 Codex: CD trigger after opening the AWOOI direct runner warn-only guard. +# 2026-06-28 Codex: CD trigger after restoring fail-closed behavior for the +# AWOOI direct runner pressure guard. # 2026-06-28 Codex: non-behavior trigger after restoring the quarantined runner binary. # 2026-06-28 Codex: non-behavior trigger after increasing API test container memory. # 2026-06-28 Codex: host 110 runner pressure remains an incident-grade guard.