fix(recovery): gate registry image pull after reboot
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 30s
CD Pipeline / build-and-deploy (push) Failing after 8s
CD Pipeline / post-deploy-checks (push) Has been skipped
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 30s
CD Pipeline / build-and-deploy (push) Failing after 8s
CD Pipeline / post-deploy-checks (push) Has been skipped
This commit is contained in:
@@ -1,3 +1,32 @@
|
||||
## 2026-06-30 — 17:57 P0-006 全主機重啟實測與受控恢復
|
||||
|
||||
**實測結論**:
|
||||
- 2026-06-30 17:16:40 CST 起觀測 110 / 120 / 121 / 188 全主機重啟,10 分鐘期限 17:26:40;期限後 Gitea / Stock edge / AWOOI VIP / public AWOOI / public Stock / 188 Ollama 外部讀回仍未全部恢復,`10_minute_auto_recovery=false`。
|
||||
- 主阻塞不是單一頁面噪音:110 Docker 因多筆 corrupt container metadata 卡在 startup,導致 registry `192.168.0.110:5000`、Harbor / registry public route、AWOOI image pull、Stock edge 與 110 SSH readback 連鎖失敗。
|
||||
|
||||
**已執行的 controlled recovery**:
|
||||
- 110 Docker corrupt metadata 只做 quarantine,不刪除:receipt `/tmp/awoooi-docker-metadata-quarantine-20260630-173012.txt`,9 個壞 metadata container dir 移至 `/var/lib/docker/containers/.awoooi-corrupt-metadata-quarantine-20260630-173012`;Docker 於 17:37:12 回 `active`,Gitea / Prometheus / Alertmanager 回通。
|
||||
- AWOOI `awoooi-prod` 因 registry `:5000` refused 進入 `ImagePullBackOff`;已驗證 120 有 `7890778b830bd0813a465931bba54feec799eeda` API/Web image cache、121 無 cache,受控 patch API/Web/Worker/Canary:`imagePullPolicy=IfNotPresent`、`nodeSelector=kubernetes.io/hostname=mon`,並暫停 API/Web/Worker template topology spread,讓 120 cache fallback 先恢復服務。
|
||||
- AWOOI runtime readback:`awoooi-api=2/2`、`awoooi-web=2/2`、`awoooi-worker=1/1`、`awoooi-auto-repair-canary=1/1`,`bad_pods=0`;VIP 內部讀回 API 200 / Web 307,public `https://awoooi.wooo.work/api/v1/health` 200、首頁 200。
|
||||
- StockPlatform public edge / API 已恢復:`https://stock.wooo.work/healthz=200`、`/api/healthz=200`;但 freshness / ingestion 仍 `status=not_configured`,blocker `postgres_not_ready`。
|
||||
|
||||
**source 修正**:
|
||||
- `full-stack-cold-start-check.sh` 新增 registry 外部 `/v2/` gate 與 K3s `IMAGE_PULL_BLOCKED` / `REGISTRY_PULL_REFUSED_EVENTS` blocker。
|
||||
- `full-stack-recovery-scorecard.sh` 新增 `CORE_REGISTRY_HTTPS_CODE`、`CORE_REGISTRY_HTTP_CODE`、`CORE_REGISTRY_READY`、`CORE_REGISTRY_BLOCKER`,registry `:5000` 未就緒時不允許 `CORE_READY`。
|
||||
|
||||
**目前 blocker / rollback**:
|
||||
- `192.168.0.110:5000` registry 仍 refused,Harbor / registry / SignOz public route 仍 502;110 SSH readback 仍 timeout,offsite evidence readback 仍 `ssh_control_channel_timeout`。
|
||||
- 120 keepalived 上有 VIP,但外部打 120/VIP NodePort refused;121 NodePort 對外 200。嘗試受控 failover 被 `sudo: a password is required` 擋住,未變更 keepalived。rollback / 正規化路徑:registry 恢復後移除 AWOOI cache fallback nodeSelector、恢復 topology spread、imagePullPolicy 回 `Always`。
|
||||
- 188 核心服務已活:Docker / Nginx / Ollama / PostgreSQL / Redis active,Ollama 本機 `/api/tags` 200;外部 `188:11434` refused 是 `127.0.0.1` 綁定行為,需依既有暴露策略判定。
|
||||
|
||||
**驗證**:
|
||||
- `pytest -q scripts/reboot-recovery/tests/test_cold_start_monitor_bounded_probes.py`:`4 passed`。
|
||||
- `bash -n scripts/reboot-recovery/full-stack-cold-start-check.sh scripts/reboot-recovery/full-stack-recovery-scorecard.sh`:通過。
|
||||
- `python3 ops/runner/guard-gitea-runner-pressure.py --root .`:`workflow_files=11 scheduled_workflows=3 auto_branch_events_on_110=0 generic_runner_labels=0`。
|
||||
- `SSH_COMMAND_TIMEOUT_SECONDS=8 bash scripts/reboot-recovery/full-stack-cold-start-check.sh --monitor-read-only --no-color`:`PASS=64 WARN=6 BLOCKED=6`,核心 AWOOI workload / public routes 已綠,registry / Harbor / SignOz / 110 readback 仍紅。
|
||||
|
||||
**邊界**:未使用 GitHub / `gh` / GitHub API,未讀 secret / token / `.env` / raw sessions / SQLite / auth,未重啟主機,未 node drain,未 firewall cutover,未 DB destructive operation。
|
||||
|
||||
## 2026-06-30 — 16:48 P0-006 110 control-channel / Stock API runtime blocker 收斂
|
||||
|
||||
**照主線處理的問題**:
|
||||
|
||||
Reference in New Issue
Block a user