docs(awooop): record t143 probe and cd evidence repair [skip ci]

This commit is contained in:
Your Name
2026-05-24 11:25:19 +08:00
parent 7211d0b7f2
commit 06dfdf7ead
2 changed files with 120 additions and 0 deletions

View File

@@ -2665,6 +2665,13 @@ Phase 6 完成後
- 判讀T135 已把 runner ownership 從雙 runner 搶工收斂到 host runner 單一主控;下一段不要重新啟用 Docker-wrapped runner而是做 runner pool / repo label 隔離、API image `apt-get` / `chown -R` 分層、Web build cache/offload、Playwright apt source-list hygiene。
- 目前進度更新AwoooP 告警可觀測鏈約 99.998%Incident-level source correlation 可見性約 98.8%Source correlation apply 狀態鏈可驗證性約 99.72%Source correlation freshness / rolling gate 約 98.2%;前端 AI 自動化管理介面同步約 99.999%Dashboard snapshot / SSE console noise 收斂約 99.2%CI/CD runner hygiene 約 99.2%Runner ownership 收斂約 96%Build host pressure治理約 82%;完整 AI 自動化管理產品化約 99.960%。
**T143 production API probe / CD evidence gate repair2026-05-24 台北)**
- 觸發production 首頁出現 `API 離線` / `--` / 進度資料不動。Live 查證 `awoooi-api` CrashLoop根因是 K8s probes 打重型 `/api/v1/health`;該 endpoint 等 `ollama` provider healthGCP-A `http://34.143.170.20:11434` timeout 後超過 probe timeoutkubelet 反覆殺 pod。第一版 GitOps commit `8558ac2d` 又被 CD `Inject K8s Secrets` 擋住,因 CD 固定走 120而 120 live 為 `NotReady,SchedulingDisabled`、SSH/API unreachable。
- 修正:`k8s/awoooi-prod/06-deployment-api.yaml` 改用 `/api/v1/health/live` / `/api/v1/health/ready` 輕量 probes`maxUnavailable=1` 解開舊 CrashLoop RS 卡 slot`.gitea/workflows/cd.yaml` 改走 Ready control-plane 121ArgoCD gate 改成必須 Synced 到目標 revisiontop-level `health=Degraded` 只記 rollout risk真正 hard gate 交給 `kubectl rollout status` 與 API healthpost-deploy notification 改讀 step outputs`scripts/alert_chain_smoke_test.py` 將 API Health 拆成核心 `api/postgresql/redis` 必須 UPAI provider例如 `ollama`)降級為 warning evidence不再讓 deploy 假紅燈。`apps/api/tests/test_alert_chain_smoke_metric.py` 補 provider-only degraded pass / core down fail 測試。
- Verificationlocal `py_compile` / `python3 -m unittest apps/api/tests/test_alert_chain_smoke_metric.py`13 tests OK/ workflow YAML parse / `git diff --check` pass。Live patch 先救回 API 2/2GitOps commit `8558ac2d` 後,`abcca652` 修 CD host`19de8345` 修 ArgoCD gate`22a4b44a` 修 Alert Chain provider warningCD #2859 success#2861 success重複 workflow_dispatch #2863 也 successdeploy marker `7211d0b7 chore(cd): deploy 22a4b44 [skip ci]`。Production API/Web/Worker/AutoRepairCanary 均 running at `22a4b44a...`public `/api/v1/health` HTTP 200core components upstatus 仍 degraded only because `ollama=down timeout`post-deploy smoke 顯示 `✅ [API Health] 核心組件 UP非阻塞降級: ollama`homepage browser readback navigation visible、API offline text absent、active events 477、service health 92%、automation evidence chain visible、console errors empty。
- 判讀T143 解的是 API probe / CD gate / post-deploy truthfulness不是把 provider timeout 消音。`ollama` timeout 現在正確落在 AI provider / router health 待辦ArgoCD top-level `health=Degraded` 仍需 T144 查清;首頁仍出現 `Executor --` / `PlayBook --` / `Ansible 未使用`,代表 execution evidence / Ansible attribution 未補齊不是前端空資料。120 control-plane 仍是 live 紅燈CD 暫時改走 121 不等於 120 已修復。Gitea workflow_dispatch duplicate run 無法由目前 API cancel也列 runner governance 技術債。
- 目前進度更新AwoooP 告警可觀測鏈約 99.998%Incident-level source correlation 可見性約 98.8%Source correlation apply 狀態鏈可驗證性約 99.72%Source correlation freshness / rolling gate 約 98.2%;前端 AI 自動化管理介面同步約 99.999%Dashboard snapshot / SSE console noise 收斂約 99.2%CI/CD runner hygiene 約 99.55%Runner ownership 收斂約 96%Runner pool inventory 約 90%Workflow label matrix 約 85%Runner isolation readiness 約 80%API probe / homepage data availability 約 96%CD control-plane resilience 約 82%Deploy rollout-risk 可觀測性約 96%CI/CD evidence 前端可見性約 94%Pipeline stage 可觀測性約 92%AI provider health / fallback truth 約 45%Execution evidence / Ansible attribution 約 55%;完整 AI 自動化管理產品化約 99.970%。
**T142 runner isolation readiness gate2026-05-24 台北)**
- 觸發T141 確認 shared runner 問題是同一個 110 `gitea-act-runner-host.service` 同時背 AWOOI、EwoooC 與 shared labels。要真正隔離前必須知道是否已有 split runner registration / service 能接手。
- 修正:新增 `ops/runner/check-runner-isolation-readiness.sh`,只讀檢查 primary runner service、runner dir、`.runner` registration file、primary label owner classes、預期 split dirs/services、active action containers輸出 `isolation_ready``blocker``safe_next_step``ops/runner/README.md` 補第八層 runner isolation readiness。