fix(recovery): gate registry image pull after reboot
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 30s
CD Pipeline / build-and-deploy (push) Failing after 8s
CD Pipeline / post-deploy-checks (push) Has been skipped

This commit is contained in:
Your Name
2026-06-30 17:59:19 +08:00
parent ae2ff6e537
commit c9ef44e050
4 changed files with 132 additions and 2 deletions

View File

@@ -1,3 +1,32 @@
## 2026-06-30 — 17:57 P0-006 全主機重啟實測與受控恢復
**實測結論**
- 2026-06-30 17:16:40 CST 起觀測 110 / 120 / 121 / 188 全主機重啟10 分鐘期限 17:26:40期限後 Gitea / Stock edge / AWOOI VIP / public AWOOI / public Stock / 188 Ollama 外部讀回仍未全部恢復,`10_minute_auto_recovery=false`
- 主阻塞不是單一頁面噪音110 Docker 因多筆 corrupt container metadata 卡在 startup導致 registry `192.168.0.110:5000`、Harbor / registry public route、AWOOI image pull、Stock edge 與 110 SSH readback 連鎖失敗。
**已執行的 controlled recovery**
- 110 Docker corrupt metadata 只做 quarantine不刪除receipt `/tmp/awoooi-docker-metadata-quarantine-20260630-173012.txt`9 個壞 metadata container dir 移至 `/var/lib/docker/containers/.awoooi-corrupt-metadata-quarantine-20260630-173012`Docker 於 17:37:12 回 `active`Gitea / Prometheus / Alertmanager 回通。
- AWOOI `awoooi-prod` 因 registry `:5000` refused 進入 `ImagePullBackOff`;已驗證 120 有 `7890778b830bd0813a465931bba54feec799eeda` API/Web image cache、121 無 cache受控 patch API/Web/Worker/Canary`imagePullPolicy=IfNotPresent``nodeSelector=kubernetes.io/hostname=mon`,並暫停 API/Web/Worker template topology spread讓 120 cache fallback 先恢復服務。
- AWOOI runtime readback`awoooi-api=2/2``awoooi-web=2/2``awoooi-worker=1/1``awoooi-auto-repair-canary=1/1``bad_pods=0`VIP 內部讀回 API 200 / Web 307public `https://awoooi.wooo.work/api/v1/health` 200、首頁 200。
- StockPlatform public edge / API 已恢復:`https://stock.wooo.work/healthz=200``/api/healthz=200`;但 freshness / ingestion 仍 `status=not_configured`blocker `postgres_not_ready`
**source 修正**
- `full-stack-cold-start-check.sh` 新增 registry 外部 `/v2/` gate 與 K3s `IMAGE_PULL_BLOCKED` / `REGISTRY_PULL_REFUSED_EVENTS` blocker。
- `full-stack-recovery-scorecard.sh` 新增 `CORE_REGISTRY_HTTPS_CODE``CORE_REGISTRY_HTTP_CODE``CORE_REGISTRY_READY``CORE_REGISTRY_BLOCKER`registry `:5000` 未就緒時不允許 `CORE_READY`
**目前 blocker / rollback**
- `192.168.0.110:5000` registry 仍 refusedHarbor / registry / SignOz public route 仍 502110 SSH readback 仍 timeoutoffsite evidence readback 仍 `ssh_control_channel_timeout`
- 120 keepalived 上有 VIP但外部打 120/VIP NodePort refused121 NodePort 對外 200。嘗試受控 failover 被 `sudo: a password is required` 擋住,未變更 keepalived。rollback / 正規化路徑registry 恢復後移除 AWOOI cache fallback nodeSelector、恢復 topology spread、imagePullPolicy 回 `Always`
- 188 核心服務已活Docker / Nginx / Ollama / PostgreSQL / Redis activeOllama 本機 `/api/tags` 200外部 `188:11434` refused 是 `127.0.0.1` 綁定行為,需依既有暴露策略判定。
**驗證**
- `pytest -q scripts/reboot-recovery/tests/test_cold_start_monitor_bounded_probes.py``4 passed`
- `bash -n scripts/reboot-recovery/full-stack-cold-start-check.sh scripts/reboot-recovery/full-stack-recovery-scorecard.sh`:通過。
- `python3 ops/runner/guard-gitea-runner-pressure.py --root .``workflow_files=11 scheduled_workflows=3 auto_branch_events_on_110=0 generic_runner_labels=0`
- `SSH_COMMAND_TIMEOUT_SECONDS=8 bash scripts/reboot-recovery/full-stack-cold-start-check.sh --monitor-read-only --no-color``PASS=64 WARN=6 BLOCKED=6`,核心 AWOOI workload / public routes 已綠registry / Harbor / SignOz / 110 readback 仍紅。
**邊界**:未使用 GitHub / `gh` / GitHub API未讀 secret / token / `.env` / raw sessions / SQLite / auth未重啟主機未 node drain未 firewall cutover未 DB destructive operation。
## 2026-06-30 — 16:48 P0-006 110 control-channel / Stock API runtime blocker 收斂
**照主線處理的問題**