docs(awooop): record t143 probe and cd evidence repair [skip ci]
This commit is contained in:
@@ -2665,6 +2665,13 @@ Phase 6 完成後
|
||||
- 判讀:T135 已把 runner ownership 從雙 runner 搶工收斂到 host runner 單一主控;下一段不要重新啟用 Docker-wrapped runner,而是做 runner pool / repo label 隔離、API image `apt-get` / `chown -R` 分層、Web build cache/offload、Playwright apt source-list hygiene。
|
||||
- 目前進度更新:AwoooP 告警可觀測鏈約 99.998%;Incident-level source correlation 可見性約 98.8%;Source correlation apply 狀態鏈可驗證性約 99.72%;Source correlation freshness / rolling gate 約 98.2%;前端 AI 自動化管理介面同步約 99.999%;Dashboard snapshot / SSE console noise 收斂約 99.2%;CI/CD runner hygiene 約 99.2%;Runner ownership 收斂約 96%;Build host pressure治理約 82%;完整 AI 自動化管理產品化約 99.960%。
|
||||
|
||||
**T143 production API probe / CD evidence gate repair(2026-05-24 台北)**:
|
||||
- 觸發:production 首頁出現 `API 離線` / `--` / 進度資料不動。Live 查證 `awoooi-api` CrashLoop,根因是 K8s probes 打重型 `/api/v1/health`;該 endpoint 等 `ollama` provider health,GCP-A `http://34.143.170.20:11434` timeout 後超過 probe timeout,kubelet 反覆殺 pod。第一版 GitOps commit `8558ac2d` 又被 CD `Inject K8s Secrets` 擋住,因 CD 固定走 120,而 120 live 為 `NotReady,SchedulingDisabled`、SSH/API unreachable。
|
||||
- 修正:`k8s/awoooi-prod/06-deployment-api.yaml` 改用 `/api/v1/health/live` / `/api/v1/health/ready` 輕量 probes,`maxUnavailable=1` 解開舊 CrashLoop RS 卡 slot;`.gitea/workflows/cd.yaml` 改走 Ready control-plane 121;ArgoCD gate 改成必須 Synced 到目標 revision,top-level `health=Degraded` 只記 rollout risk,真正 hard gate 交給 `kubectl rollout status` 與 API health;post-deploy notification 改讀 step outputs;`scripts/alert_chain_smoke_test.py` 將 API Health 拆成核心 `api/postgresql/redis` 必須 UP,AI provider(例如 `ollama`)降級為 warning evidence,不再讓 deploy 假紅燈。`apps/api/tests/test_alert_chain_smoke_metric.py` 補 provider-only degraded pass / core down fail 測試。
|
||||
- Verification:local `py_compile` / `python3 -m unittest apps/api/tests/test_alert_chain_smoke_metric.py`(13 tests OK)/ workflow YAML parse / `git diff --check` pass。Live patch 先救回 API 2/2;GitOps commit `8558ac2d` 後,`abcca652` 修 CD host,`19de8345` 修 ArgoCD gate,`22a4b44a` 修 Alert Chain provider warning;CD #2859 success,#2861 success,重複 workflow_dispatch #2863 也 success;deploy marker `7211d0b7 chore(cd): deploy 22a4b44 [skip ci]`。Production API/Web/Worker/AutoRepairCanary 均 running at `22a4b44a...`;public `/api/v1/health` HTTP 200,core components up,status 仍 degraded only because `ollama=down timeout`;post-deploy smoke 顯示 `✅ [API Health] 核心組件 UP;非阻塞降級: ollama`;homepage browser readback navigation visible、API offline text absent、active events 477、service health 92%、automation evidence chain visible、console errors empty。
|
||||
- 判讀:T143 解的是 API probe / CD gate / post-deploy truthfulness,不是把 provider timeout 消音。`ollama` timeout 現在正確落在 AI provider / router health 待辦;ArgoCD top-level `health=Degraded` 仍需 T144 查清;首頁仍出現 `Executor --` / `PlayBook --` / `Ansible 未使用`,代表 execution evidence / Ansible attribution 未補齊,不是前端空資料。120 control-plane 仍是 live 紅燈,CD 暫時改走 121 不等於 120 已修復。Gitea workflow_dispatch duplicate run 無法由目前 API cancel,也列 runner governance 技術債。
|
||||
- 目前進度更新:AwoooP 告警可觀測鏈約 99.998%;Incident-level source correlation 可見性約 98.8%;Source correlation apply 狀態鏈可驗證性約 99.72%;Source correlation freshness / rolling gate 約 98.2%;前端 AI 自動化管理介面同步約 99.999%;Dashboard snapshot / SSE console noise 收斂約 99.2%;CI/CD runner hygiene 約 99.55%;Runner ownership 收斂約 96%;Runner pool inventory 約 90%;Workflow label matrix 約 85%;Runner isolation readiness 約 80%;API probe / homepage data availability 約 96%;CD control-plane resilience 約 82%;Deploy rollout-risk 可觀測性約 96%;CI/CD evidence 前端可見性約 94%;Pipeline stage 可觀測性約 92%;AI provider health / fallback truth 約 45%;Execution evidence / Ansible attribution 約 55%;完整 AI 自動化管理產品化約 99.970%。
|
||||
|
||||
**T142 runner isolation readiness gate(2026-05-24 台北)**:
|
||||
- 觸發:T141 確認 shared runner 問題是同一個 110 `gitea-act-runner-host.service` 同時背 AWOOI、EwoooC 與 shared labels。要真正隔離前,必須知道是否已有 split runner registration / service 能接手。
|
||||
- 修正:新增 `ops/runner/check-runner-isolation-readiness.sh`,只讀檢查 primary runner service、runner dir、`.runner` registration file、primary label owner classes、預期 split dirs/services、active action containers,輸出 `isolation_ready`、`blocker`、`safe_next_step`。`ops/runner/README.md` 補第八層 runner isolation readiness。
|
||||
|
||||
Reference in New Issue
Block a user