fix(recovery): reopen controlled drain lane source

This commit is contained in:
Your Name
2026-06-28 19:28:37 +08:00
parent b1da5cc7d3
commit d5c9d467c1
17 changed files with 766 additions and 1051 deletions

View File

@@ -406,41 +406,28 @@ Gitea service 名稱。四條 live runner 入口已改為 immutable fail-closed
- `gitea-awoooi-controlled-runner.service`
- `gitea-act-runner-awoooi-open.service`
`awoooi-cd-lane.service` `awoooi-cd-lane-drain.service` 目前同屬 110 壓力事故保護面。
未完成 runner 搬遷或非 110 硬限流前,不得用 sentinel、`START_CONTROLLED_CD_LANE`
quarantine restore source 或 `systemd-run` 讓它們恢復 active。
`awoooi-cd-lane.service` 是專用 controlled lane不屬於 legacy runner mask 清單;
只有在 `/run/awoooi-cd-lane-enabled``AWOOOI_START_CONTROLLED_CD_LANE=1`
存在、`capacity=1`、label 僅限 `awoooi-ubuntu` / `awoooi-host`、沒有
`ubuntu-latest` / StockPlatform / headless / Playwright 類泛用重型 label
systemd CPU / memory / tasks 限流、root restore-source left `0`
post-apply verifier 可讀回 `CD_LANE_CONTROLLED ok=1` 時,才可受控恢復。
未滿足條件時 cd-lane 應回到 static `/bin/false` unit 與 shell stub。
2026-06-28 fail-closed enforcer updatesource of truth 為:
未完成 runner 搬遷、限流、smoke 排程前,不得解除 legacy mask、恢復泛用 runner label
或把 host pressure gate 預設改成 warn-only。
- `scripts/reboot-recovery/enforce-110-runner-failclosed.sh`
- `ops/runner/awoooi-runner-failclosed-enforcer.service`
- `ops/runner/awoooi-runner-failclosed-enforcer.timer`
- `ops/runner/awoooi-runner-failclosed-authority.service`
- `ops/runner/awoooi-runner-failclosed-authority.timer`
2026-06-28 controlled update舊的 manual-only / freeze guard 已改為分流判讀。
legacy runner 仍維持 masked / fail-closed;專用 `awoooi-cd-lane.service`
`awoooi-cd-lane-drain.service` 只要通過 capacity、label、binary、process 與
systemd limit、root restore-source left `0`、post-apply verifier可作為
AWOOOI 專用受控部署 lane。
live 110 必須安裝 authority copy `/usr/local/lib/awoooi/enforce-110-runner-failclosed.authority.sh`
與 canonical `/usr/local/lib/awoooi/enforce-110-runner-failclosed.sh`cron / systemd authority 一律執行
authority copy讓外部 opener 覆寫 canonical 時仍可自動修復。
`/usr/local/bin/awoooi-enforce-runner-failclosed-110.sh` 只作相容 wrapper。必須啟用
`awoooi-runner-failclosed-enforcer.timer``awoooi-runner-failclosed-authority.timer`
`/etc/cron.d/awoooi-runner-failclosed-authority` 必須存在,作為 systemd timers 被短命外部 opener mask 掉時的第三層收斂 authority。
cold-start、post-start 與 P3 verifier 必須讀回兩個 timer 都 `active` / `enabled`
兩個 service 都 `Result=success`、runner / lane units
全部 masked / inactive、process `0`、active job container `0`、root restore-source left `0`
若外部 opener 暫時把 unit 恢復成 `ConditionPathExists=/run/awoooi-runner-migrated-or-hard-limited`
的 fail-closed stubverifier 可視為 sealed fallbackenforcer 下一輪仍需收斂回 masked / inactive。
`/tmp/enforce-110-runner-failclosed.sh``/tmp/awoooi-enforce-runner-failclosed-110.sh*`、舊 cd-lane unit template、startup runner-open
drop-in、systemd unit backup、`awoooi-runner-failclosed-opened-*`
`awoooi-runner-failclosed-*-opened-*``awoooi-runner-failclosed-quarantine-*``failclosed-final-mask-*` disabler artifact、
root live artifact 與 lane registration 檔名都屬 restore source
必須由 enforcer 封存或改成 fail-closed stub不得保留舊 `.codex` enforcer source 讓 drain lane
復活。
未完成 runner 搬遷、硬限流、smoke 排程前,不得解除 mask、恢復泛用 runner label、
恢復 cd-lane / drain ELF或把 host pressure gate 預設改成 warn-only所有會命中
`awoooi-ubuntu` / `awoooi-host``.gitea/workflows` 都不得保留 `push`
`pull_request``pull_request_target` 自動事件。
若 verifier 失敗rollback 回 inactive / masked / fail-closed stub若 verifier
通過,不得再用 generic runner fail-closed 規則殺掉 controlled lane。legacy / generic
runner 仍不得解除 mask 或恢復泛用 label所有會命中 `awoooi-ubuntu` / `awoooi-host`
`.gitea/workflows` 都不得保留 `push``pull_request``pull_request_target`
自動事件
### 第九層修復: workflow pressure source guard
@@ -458,6 +445,8 @@ python3 ops/runner/guard-gitea-runner-pressure.py --root .
`scripts/ops/ansible-validate.sh` 會執行同一 guard。若要恢復自動事件必須先有
runner 搬遷或非 110 硬限流的 source-of-truth diff、rollback 與 post-apply verifier。
`cd.yaml` / `code-review.yaml` 不得因非事故級 guard 長期停在 `workflow_dispatch` only
恢復自動事件前必須先通過 runner 搬遷或非 110 硬限流 verifier。
---
版本: v2.0 | 更新: 2026-03-29 | 作者: Claude Code

View File

@@ -1,10 +0,0 @@
[Unit]
Description=AWOOOI 110 runner/CD lane fail-closed authority
Documentation=file:/usr/local/lib/awoooi/enforce-110-runner-failclosed.authority.sh
Wants=network-online.target
After=network-online.target docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/lib/awoooi/enforce-110-runner-failclosed.authority.sh --apply
TimeoutStartSec=180

View File

@@ -1,12 +0,0 @@
[Unit]
Description=Run AWOOOI 110 runner/CD lane fail-closed authority
[Timer]
OnBootSec=20s
OnUnitInactiveSec=20s
AccuracySec=5s
Persistent=true
Unit=awoooi-runner-failclosed-authority.service
[Install]
WantedBy=timers.target

View File

@@ -1,10 +0,0 @@
[Unit]
Description=AWOOOI 110 runner/CD lane fail-closed enforcer
Documentation=file:/usr/local/lib/awoooi/enforce-110-runner-failclosed.authority.sh
Wants=network-online.target
After=network-online.target docker.service
[Service]
Type=oneshot
ExecStart=/usr/local/lib/awoooi/enforce-110-runner-failclosed.authority.sh --apply
TimeoutStartSec=180

View File

@@ -1,12 +0,0 @@
[Unit]
Description=Run AWOOOI 110 runner/CD lane fail-closed enforcer
[Timer]
OnBootSec=30s
OnUnitInactiveSec=120s
AccuracySec=15s
Persistent=true
Unit=awoooi-runner-failclosed-enforcer.service
[Install]
WantedBy=timers.target