diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index c884119e3..1228f2499 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -31,17 +31,19 @@ - ArgoCD refresh 後成功同步到 `9db4f72cb`,但 API rollout 卡在 1/2;Pending pod event 顯示 `nodeSelector=mon` 搭配 `topologySpreadConstraints minDomains=2 whenUnsatisfiable=DoNotSchedule` 自相矛盾,造成 scheduler deadlock。 - 已把 API / Web / Worker 的 topology spread 從硬性 `DoNotSchedule + minDomains=2` 調整為柔性 `ScheduleAnyway`,保留 hostname spread preference,但不再阻塞 post-reboot / GitOps rollout。 - API 新 pod 排程後又 `CrashLoopBackOff`;previous log 顯示 `harbor_registry_controlled_recovery_preflight.py` 在 production container layout `/app/src/services/...` 使用 `Path(__file__).parents[4]` 造成 `IndexError: 4`。已改用既有 `resolve_repo_root(Path(__file__))`,並同步修掉同型的 StockPlatform controlled recovery preflight 路徑解析,避免下一個 import 再炸。 +- 最新 API image `54ac3cd...` 啟動後又因 `asyncpg.exceptions.TooManyConnectionsError: too many connections for role "awoooi"` CrashLoop;readback 顯示 Postgres `max_connections=300`,但 `awoooi` role `rolconnlimit=2`,且 `awoooi_migrator` 無 `ALTER ROLE` 權限。非 secret admin path 目前不可用,因此先把 API `replicas` 暫時降為 `1`,讓最新版 API 在既有 role limit 內恢復;`replicas=2` 必須等 DB role limit / pool budget 修復後再恢復。 **驗證**: - `argocd-repo-server` rollout:`deployment "argocd-repo-server" successfully rolled out`。 - ArgoCD readback:`sync=Synced rev=9db4f72cbf98`,API deployment image / env 已更新為 `06819ea96c05...`。 - `kubectl apply --dry-run=server` 驗證 `awoooi-api`、`awoooi-web`、`awoooi-worker` manifests:通過。 - `py_compile` / targeted pytest 需在 path resolver fix 後重跑並推正常 CD,因為 production 必須建出新 API image 才能離開 CrashLoop。 +- DB role readback:`awoooi rolconnlimit=2`、`awoooi_migrator rolconnlimit=-1` 但無 `ALTER ROLE` 權限;188 local postgres admin path 需要密碼,未讀 secret。 **邊界**:未讀 secret / token / `.env` / raw sessions / SQLite / auth;未使用 GitHub / `gh` / GitHub API;未重啟主機 / Docker / Nginx / K3s / DB / firewall;live patch 僅限 ArgoCD repo-server init command 與 prod Deployment topology scheduler policy。 **下一步**: -- 推 Gitea main 後等 CD 建新 API image,讀回 API / Web / Worker 全部 Ready、production workbench runtime/desired tag 等於最新 commit,再判定 Gitea/CD 恢復完成。 +- 推 Gitea main 後等 ArgoCD 套用 `replicas=1` recovery cap,讀回 API / Web / Worker Ready、production workbench runtime/desired tag 等於最新 deploy marker;後續 P0 必須修 DB role limit / pool budget 後恢復 API `replicas=2`。 ## 2026-07-01 — 15:25 Gitea live recovery after 110 Docker/control-plane failure diff --git a/k8s/awoooi-prod/06-deployment-api.yaml b/k8s/awoooi-prod/06-deployment-api.yaml index 2cec4f762..135b3c56f 100644 --- a/k8s/awoooi-prod/06-deployment-api.yaml +++ b/k8s/awoooi-prod/06-deployment-api.yaml @@ -14,7 +14,11 @@ metadata: system: awoooi environment: prod spec: - replicas: 2 + # 2026-07-01 Codex: temporary post-reboot recovery cap. The production + # database role `awoooi` currently has CONNECTION LIMIT 2, so a two-replica + # API rollout deadlocks when the old pod still holds DB connections. Restore + # replicas=2 only after the DB role limit / pool budget is raised and verified. + replicas: 1 revisionHistoryLimit: 3 selector: matchLabels: