From 06dfdf7ead9a985b896cb41f5d643e1cef78ccdb Mon Sep 17 00:00:00 2001 From: Your Name Date: Sun, 24 May 2026 11:25:19 +0800 Subject: [PATCH] docs(awooop): record t143 probe and cd evidence repair [skip ci] --- docs/LOGBOOK.md | 113 ++++++++++++++++++ ...-04-15-MASTER-ai-autonomous-flywheel-v2.md | 7 ++ 2 files changed, 120 insertions(+) diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index c871b5000..bf57e96bd 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,116 @@ +## 2026-05-24｜T143 production API probe / CD evidence gate repair + +**觸發**： + +- Production 首頁一度顯示 `API 離線` / `--` / 小龍蝦進度資料不動；live 查證 `awoooi-api` 在 K8s 反覆 CrashLoop。 +- 根因不是前端壞掉，而是 K8s startup/readiness/liveness 都打重型 `/api/v1/health`；該 endpoint 會等 `ollama` provider health，當 GCP-A `http://34.143.170.20:11434` timeout 時超過 probe `timeoutSeconds=5`，kubelet 就把 pod 殺掉。 +- 第一版 GitOps commit `8558ac2d` 被 CD 擋在 `Inject K8s Secrets`，因 `.gitea/workflows/cd.yaml` 固定 `K8S_SSH_HOST=192.168.0.120`，但 120 live 是 `NotReady,SchedulingDisabled` 且 SSH/API endpoint 不通；121 才是 Ready control-plane。 + +**修正**： + +- `k8s/awoooi-prod/06-deployment-api.yaml`： + - liveness 改 `/api/v1/health/live`。 + - readiness 改 `/api/v1/health/ready`。 + - startup 改 `/api/v1/health/live`。 + - rolling update `maxUnavailable` 改 `1`，避免舊 CrashLoop ReplicaSet 佔住 slot 時 rollout 卡死。 +- `.gitea/workflows/cd.yaml`： + - `K8S_SSH_HOST` / `K8S_API_SERVER` 從 120 改為 121。 + - ArgoCD gate 改為必須 `Synced` 到目標 revision；top-level `Application.health=Degraded` 只記 rollout risk，真正 pass/fail 交給 `kubectl rollout status` 與 API health。 + - API health wait 加寬 `--max-time`，避免剛 rollout 的短暫 provider timeout 被誤記成部署失敗。 + - post-deploy 通知改讀 step outputs，不再只看 step outcome，避免 Alert Chain 實際 fail 但 summary 假綠燈。 +- `scripts/alert_chain_smoke_test.py`： + - API Health smoke 改成核心組件 `api/postgresql/redis` 必須 UP。 + - `ollama` 等 AI provider 降級改為非阻塞 warning evidence，交給 AI Router / provider health 治理，不讓 CD 假紅燈。 +- `apps/api/tests/test_alert_chain_smoke_metric.py` 補兩個 unit tests：provider-only degraded 應 pass；core component down 應 fail。 + +**Verification**： + +```text +Local: +python3 -m py_compile scripts/alert_chain_smoke_test.py -> pass +python3 -m unittest apps/api/tests/test_alert_chain_smoke_metric.py -> 13 tests OK +ruby YAML.load_file(".gitea/workflows/cd.yaml") -> yaml ok +git diff --check -> pass + +Live incident repair: +kubectl patch deployment/awoooi-api probes -> live API recovered +kubectl patch maxUnavailable=1 -> rollout unblocked +awoooi-api -> 2/2 ready + +Gitea / CD: +8558ac2d fix(k8s): use lightweight api probes + #2851 CD failed at Inject K8s Secrets because 120 SSH/API were unreachable. +abcca652 fix(cd): use ready k8s control plane + #2854 dispatched, secrets injected, then cancelled by next run after ArgoCD health gate fix. +19de8345 fix(cd): gate deploy on synced revision + #2859 success; deploy marker f3b85cda; build-and-deploy / post-deploy success. +22a4b44a fix(ci): report provider degradation as warning + #2861 success; #2863 duplicate workflow_dispatch also success. + deploy marker 7211d0b7. + +Production K8s: +awoooi-api 2/2 image 192.168.0.110:5000/awoooi/api:22a4b44a... +awoooi-web 2/2 image 192.168.0.110:5000/awoooi/web:22a4b44a... +awoooi-worker 1/1 image 192.168.0.110:5000/awoooi/api:22a4b44a... +awoooi-auto-repair-canary 1/1 image 192.168.0.110:5000/awoooi/api:22a4b44a... + +ArgoCD: +sync=Synced +health=Degraded +revision=7211d0b7f29899de02680324c87b619fd3026ac9 + +Public health: +GET https://awoooi.wooo.work/api/v1/health -> HTTP 200 in ~5.1s +status=degraded because ollama=down timeout +core: api=up, postgresql=up, redis=up, openclaw=up, signoz=up + +Post-deploy smoke: +Alert Chain Smoke -> ✅ [API Health] 核心組件 UP；非阻塞降級: ollama +Monitoring coverage -> 100.0% +Source correlation applied-link smoke -> already_applied / verified +CI/CD success notification -> mirrored through AWOOI API + +Homepage browser readback: +https://awoooi.wooo.work/zh-TW +navigation visible=true +API offline text=false +active events=477 +service health=92% +automation evidence chain visible +console errors=[] +``` + +**判讀**： + +- Production API / homepage empty-data 問題已救回，並用 GitOps/CD 落地；不是只做 live patch。 +- `ollama` timeout 仍是真問題，但它現在被正確分類為 AI provider / router degraded，不再偽裝成 K8s API probe 或 deploy failure。 +- ArgoCD top-level `health=Degraded` 仍需後續查清，因目前 resource list 沒提供 unhealthy detail；CD 已改成先記 rollout risk，再以 rollout / public health 作 hard gate。 +- 首頁仍能看到 `Executor --`、`PlayBook --`、`Ansible 未使用`，這代表部分 Incident evidence 沒有 `automation_operation_log` / playbook / Ansible executor row，不是導航列或 API 離線。下一段應補 execution evidence ingestion 與 Ansible usage attribution。 +- 手動 workflow_dispatch `#2863` 無法用目前 Gitea API cancel，最後成為重複驗證 run；runner/concurrency/cancel governance 仍是後續技術債。 +- 120 control-plane 仍是 live 紅燈：`NotReady,SchedulingDisabled`，SSH timeout，K8s API `no route to host`；CD 暫時改走 121，不等於 120 已修復。 + +**目前整體進度**： + +- AwoooP 告警可觀測鏈：99.998%。 +- Incident-level source correlation 可見性：98.8%。 +- Source correlation apply 狀態鏈可驗證性：99.72%。 +- Source correlation freshness / rolling gate：98.2%。 +- 前端 AI 自動化管理介面同步：99.999%。 +- Dashboard snapshot / SSE console noise 收斂：99.2%。 +- CI/CD runner hygiene：99.55%。 +- Runner ownership 收斂：96%。 +- Runner pool inventory：90%。 +- Workflow label matrix：85%。 +- Runner isolation readiness：80%。 +- API probe / homepage data availability：0% → 96%。 +- CD control-plane resilience：0% → 82%。 +- Deploy rollout-risk 可觀測性：91% → 96%。 +- CI/CD evidence 前端可見性：92% → 94%。 +- Pipeline stage 可觀測性：88% → 92%。 +- AI provider health / fallback truth：45%（ollama/GCP-A timeout 仍待 T144）。 +- Execution evidence / Ansible attribution：55%。 +- 完整 AI 自動化管理產品化：99.967% → 99.970%。 + ## 2026-05-24｜T142 runner isolation readiness gate **觸發**： diff --git a/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md b/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md index 6ea2b23a7..a532ccd30 100644 --- a/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md +++ b/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md @@ -2665,6 +2665,13 @@ Phase 6 完成後 - 判讀：T135 已把 runner ownership 從雙 runner 搶工收斂到 host runner 單一主控；下一段不要重新啟用 Docker-wrapped runner，而是做 runner pool / repo label 隔離、API image `apt-get` / `chown -R` 分層、Web build cache/offload、Playwright apt source-list hygiene。 - 目前進度更新：AwoooP 告警可觀測鏈約 99.998%；Incident-level source correlation 可見性約 98.8%；Source correlation apply 狀態鏈可驗證性約 99.72%；Source correlation freshness / rolling gate 約 98.2%；前端 AI 自動化管理介面同步約 99.999%；Dashboard snapshot / SSE console noise 收斂約 99.2%；CI/CD runner hygiene 約 99.2%；Runner ownership 收斂約 96%；Build host pressure治理約 82%；完整 AI 自動化管理產品化約 99.960%。 +**T143 production API probe / CD evidence gate repair（2026-05-24 台北）**： +- 觸發：production 首頁出現 `API 離線` / `--` / 進度資料不動。Live 查證 `awoooi-api` CrashLoop，根因是 K8s probes 打重型 `/api/v1/health`；該 endpoint 等 `ollama` provider health，GCP-A `http://34.143.170.20:11434` timeout 後超過 probe timeout，kubelet 反覆殺 pod。第一版 GitOps commit `8558ac2d` 又被 CD `Inject K8s Secrets` 擋住，因 CD 固定走 120，而 120 live 為 `NotReady,SchedulingDisabled`、SSH/API unreachable。 +- 修正：`k8s/awoooi-prod/06-deployment-api.yaml` 改用 `/api/v1/health/live` / `/api/v1/health/ready` 輕量 probes，`maxUnavailable=1` 解開舊 CrashLoop RS 卡 slot；`.gitea/workflows/cd.yaml` 改走 Ready control-plane 121；ArgoCD gate 改成必須 Synced 到目標 revision，top-level `health=Degraded` 只記 rollout risk，真正 hard gate 交給 `kubectl rollout status` 與 API health；post-deploy notification 改讀 step outputs；`scripts/alert_chain_smoke_test.py` 將 API Health 拆成核心 `api/postgresql/redis` 必須 UP，AI provider（例如 `ollama`）降級為 warning evidence，不再讓 deploy 假紅燈。`apps/api/tests/test_alert_chain_smoke_metric.py` 補 provider-only degraded pass / core down fail 測試。 +- Verification：local `py_compile` / `python3 -m unittest apps/api/tests/test_alert_chain_smoke_metric.py`（13 tests OK）/ workflow YAML parse / `git diff --check` pass。Live patch 先救回 API 2/2；GitOps commit `8558ac2d` 後，`abcca652` 修 CD host，`19de8345` 修 ArgoCD gate，`22a4b44a` 修 Alert Chain provider warning；CD #2859 success，#2861 success，重複 workflow_dispatch #2863 也 success；deploy marker `7211d0b7 chore(cd): deploy 22a4b44 [skip ci]`。Production API/Web/Worker/AutoRepairCanary 均 running at `22a4b44a...`；public `/api/v1/health` HTTP 200，core components up，status 仍 degraded only because `ollama=down timeout`；post-deploy smoke 顯示 `✅ [API Health] 核心組件 UP；非阻塞降級: ollama`；homepage browser readback navigation visible、API offline text absent、active events 477、service health 92%、automation evidence chain visible、console errors empty。 +- 判讀：T143 解的是 API probe / CD gate / post-deploy truthfulness，不是把 provider timeout 消音。`ollama` timeout 現在正確落在 AI provider / router health 待辦；ArgoCD top-level `health=Degraded` 仍需 T144 查清；首頁仍出現 `Executor --` / `PlayBook --` / `Ansible 未使用`，代表 execution evidence / Ansible attribution 未補齊，不是前端空資料。120 control-plane 仍是 live 紅燈，CD 暫時改走 121 不等於 120 已修復。Gitea workflow_dispatch duplicate run 無法由目前 API cancel，也列 runner governance 技術債。 +- 目前進度更新：AwoooP 告警可觀測鏈約 99.998%；Incident-level source correlation 可見性約 98.8%；Source correlation apply 狀態鏈可驗證性約 99.72%；Source correlation freshness / rolling gate 約 98.2%；前端 AI 自動化管理介面同步約 99.999%；Dashboard snapshot / SSE console noise 收斂約 99.2%；CI/CD runner hygiene 約 99.55%；Runner ownership 收斂約 96%；Runner pool inventory 約 90%；Workflow label matrix 約 85%；Runner isolation readiness 約 80%；API probe / homepage data availability 約 96%；CD control-plane resilience 約 82%；Deploy rollout-risk 可觀測性約 96%；CI/CD evidence 前端可見性約 94%；Pipeline stage 可觀測性約 92%；AI provider health / fallback truth 約 45%；Execution evidence / Ansible attribution 約 55%；完整 AI 自動化管理產品化約 99.970%。 + **T142 runner isolation readiness gate（2026-05-24 台北）**： - 觸發：T141 確認 shared runner 問題是同一個 110 `gitea-act-runner-host.service` 同時背 AWOOI、EwoooC 與 shared labels。要真正隔離前，必須知道是否已有 split runner registration / service 能接手。 - 修正：新增 `ops/runner/check-runner-isolation-readiness.sh`，只讀檢查 primary runner service、runner dir、`.runner` registration file、primary label owner classes、預期 split dirs/services、active action containers，輸出 `isolation_ready`、`blocker`、`safe_next_step`。`ops/runner/README.md` 補第八層 runner isolation readiness。