wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

fix(recovery): bound 188 startup clawbot rebuild
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
CD Pipeline / tests (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Your Name
2026-06-30 22:37:33 +08:00
parent 51d87b237d
commit 1c3c7279ac
5 changed files with 108 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
docs/LOGBOOK.md
View File

@@ -50778,3 +50778,32 @@ production browser smoke:
**下一步**：
- commit / push 後等待 Gitea CD；若仍卡 Harbor/110，P0 主線下一步是 110 local console / root shell 執行 recovery package check-mode，讀回 `awoooi-host` controlled lane verifier、Harbor queue、registry `/v2/`、deploy marker，並把 post-apply verifier receipt 寫回 AI Loop queue / KM / RAG / MCP / PlayBook。
## 2026-06-30 — 22:42 188 startup timeout / ClawBot rebuild SLO 修復
**完成內容**：
- 讀回 22:30 live post-reboot readiness：`SERVICE_GREEN=0`、`PRODUCT_DATA_GREEN=0`、`BACKUP_CORE_GREEN=0`、`HOST_188_SERVICE_GREEN=0`、`OVERALL_DECLARATION=SERVICE_BLOCKED`；Stock freshness / ingestion 仍是 `postgres_not_ready`。
- 產生最新 SLO scorecard `/tmp/awoooi-reboot-slo-live-20260630-2231-scorecard.json`：`status=blocked_reboot_auto_recovery_slo_not_ready`、`can_claim_all_services_recovered_within_target=false`；111 unreachable，99 boot / VMware autostart unknown，110/112/120/121/188 皆已超過 10 分鐘窗口。
- 讀回 188 host hygiene：`awoooi-startup.service` 因 300 秒 timeout 失敗，卡在 ClawBot `docker compose build --no-cache`；但 live `clawbot` / `clawbot-redis` 容器已 healthy，證明舊 startup path 會把單一 rebuild 誤放大成整機 post-reboot failed。
- 修正 `scripts/reboot-recovery/awoooi-startup.sh`：ClawBot startup 改成 bounded `docker compose up -d`，rebuild 預設 opt-in (`CLAWBOT_STARTUP_REBUILD_ALLOWED=0`) 且有 timeout；移除預設 `--no-cache` rebuild。
- 修正 `scripts/reboot-recovery/awoooi-startup.service`：`TimeoutStartSec=600`，對齊 10 分鐘 SLO，但不允許單一服務 rebuild 佔滿整個恢復窗口。
- 補 `scripts/reboot-recovery/tests/test_188_host_hygiene_checklist.py`，鎖住 bounded / opt-in rebuild 與 600 秒 unit timeout。
**本地驗證結果**：
- `bash -n scripts/reboot-recovery/awoooi-startup.sh scripts/reboot-recovery/awoooi-startup.service scripts/reboot-recovery/188-host-hygiene-maintenance-checklist.sh`：通過。
- `DATABASE_URL=sqlite+aiosqlite:////tmp/awoooi-codex-api-test.db PYTHONPATH=apps/api python3.11 -m pytest scripts/reboot-recovery/tests/test_188_host_hygiene_checklist.py scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_scorecard.py ops/runner/test_cd_controlled_runtime_profile.py -q`：`42 passed`。
- `python3.11 ops/runner/guard-gitea-runner-pressure.py --root .`：通過，`auto_branch_events_on_110=0`、`generic_runner_labels=0`。
- `node scripts/ci/check-gitea-step-env-secrets.js .gitea/workflows/cd.yaml .gitea/workflows/harbor-110-local-repair.yaml`：通過。
**live truth**：
- `https://registry.wooo.work/v2/`、`http://192.168.0.110:5000/v2/`、`https://harbor.wooo.work/api/v2.0/health` 仍回 502。
- public Gitea queue 仍是 `blocked_harbor_110_repair_no_matching_runner`；CD `#4100` running，Harbor repair `#4099` Waiting，no matching label `awoooi-host`。
- Production delivery / SLO API 仍含 2026-06-29 舊資料，不可作為本輪恢復證據；本輪以 live `/tmp/awoooi-reboot-slo-live-20260630-2231-scorecard.json` 為準。
**仍維持**：
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth；沒有讀 `.runner` 內容。
- 沒有使用 GitHub / gh / GitHub API / GitHub Actions。
- 沒有重啟主機，沒有 Docker / Nginx / K3s / DB restart，沒有 workflow_dispatch，沒有 DB write / restore / prune。
**下一步**：
- commit / push 後等待 Gitea CD；若 runtime 仍卡 Harbor / 110，P0 主線仍是恢復 110 `awoooi-host` controlled lane 或 110 local recovery package，再重讀 registry `/v2/`、full-stack cold-start、Stock freshness、backup core 與 SLO scorecard。

27
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -2334,3 +2334,30 @@ All must be true:
- `stockplatform-shared-ui-monitor.timer` / service source-of-truth 仍需清理或重建；2026-06-12 只停用 stale timer 以解除 host degraded。
- 111 local Ollama fallback 目前不可達；production provider 由 GCP-A / GCP-B 承接，但 111 恢復應另列 AI provider resilience 工作。
- 本 SOP v1.5 新增內容已用繁體中文補強；舊章節仍有英文段落，後續 runbook hygiene 應分批翻譯，不要在事故 P0 中混入大規模格式重排。
### 16.1 188 startup / ClawBot rebuild 不得阻塞 SLO
2026-06-30 主機重啟後，188 `awoooi-startup.service` 在 `TimeoutStartSec=300` 內被 systemd 砍掉；live journal 顯示卡在 ClawBot `docker compose build --no-cache`，但事後 `clawbot` / `clawbot-redis` 容器已 healthy。此類情況不可再解讀為需要盲目 restart / rebuild；正確做法是讓 startup oneshot 完成，將 ClawBot 健康交給 post-reboot verifier 判斷。
新的 source-of-truth：
- `scripts/reboot-recovery/awoooi-startup.sh`
- `scripts/reboot-recovery/awoooi-startup.service`
- `scripts/reboot-recovery/tests/test_188_host_hygiene_checklist.py`
規則：
- ClawBot startup 只允許 bounded `docker compose up -d`，預設確認窗口 `CLAWBOT_UP_TIMEOUT_SECONDS=90`。
- ClawBot image rebuild 預設停用：`CLAWBOT_STARTUP_REBUILD_ALLOWED=0`。
- 只有在 maintenance / controlled recovery 明確設定 `CLAWBOT_STARTUP_REBUILD_ALLOWED=1` 時，才允許 bounded rebuild，且 `CLAWBOT_BUILD_TIMEOUT_SECONDS=120`。
- 不得在 reboot startup path 預設執行 `docker compose build --no-cache`。
- `awoooi-startup.service` timeout 對齊 10 分鐘 SLO：`TimeoutStartSec=600`。
- 若 ClawBot 未確認 running，startup script 只留下 warning；`post-reboot-readiness-summary.sh` / `full-stack-cold-start-check.sh` 必須負責把它標成服務 blocker。
驗證命令：
```bash
bash -n scripts/reboot-recovery/awoooi-startup.sh
DATABASE_URL=sqlite+aiosqlite:////tmp/awoooi-codex-api-test.db PYTHONPATH=apps/api \
python3.11 -m pytest scripts/reboot-recovery/tests/test_188_host_hygiene_checklist.py -q
```