fix(ollama): restore 111 fallback before gemini
Some checks failed
Ansible Lint / lint (push) Failing after 39s
CD Pipeline / tests (push) Successful in 56s
Code Review / ai-code-review (push) Successful in 7s
CD Pipeline / build-and-deploy (push) Successful in 3m29s
CD Pipeline / post-deploy-checks (push) Successful in 1m36s
Some checks failed
Ansible Lint / lint (push) Failing after 39s
CD Pipeline / tests (push) Successful in 56s
Code Review / ai-code-review (push) Successful in 7s
CD Pipeline / build-and-deploy (push) Successful in 3m29s
CD Pipeline / post-deploy-checks (push) Successful in 1m36s
This commit is contained in:
@@ -1,3 +1,81 @@
|
||||
## 2026-05-25|T172 Ollama Provider lane 恢復 111 fallback 接線
|
||||
|
||||
**背景**:
|
||||
|
||||
- T171 production smoke 發現 `/api/v1/health` degraded,`ai-route-status` 選到 Gemini。
|
||||
- 統帥再次確認所有 Ollama 路徑必須依序 `GCP-A → GCP-B → 111 → Gemini`,Gemini 只能是最後備援。
|
||||
- 本階段先驗 live state,不用 label 包裝成修好;目標是讓路由在 GCP-A/GCP-B upstream 掛掉時能真正落到 111。
|
||||
|
||||
**live 驗證結論**:
|
||||
|
||||
- API pod / 110 / 本機打 direct GCP-A `34.143.170.20:11434`、GCP-B `34.21.145.224:11434` 均 connection refused。
|
||||
- 110 nginx 11435/11436/11437 都有 listen,但 live 存在重複設定:
|
||||
- `/etc/nginx/conf.d/ollama-gcp-proxy.conf`
|
||||
- `/etc/nginx/sites-enabled/110-ollama-proxy.conf`
|
||||
- 111 本機 Ollama `127.0.0.1:11434` 可用;對外 `192.168.0.111:11434` 由 `com.momo.ollama111-allow-proxy` 控制。
|
||||
- 111 allowlist 原本只允許 `127.0.0.1/32,192.168.0.111/32,192.168.0.188/32`,因此 110 / K3s node 進來會被 reset。
|
||||
- live NetworkPolicy 原本只允許 Pod → 110 `11435/11436`,缺 111 fallback proxy `11437`。
|
||||
|
||||
**本次修補**:
|
||||
|
||||
- `k8s/awoooi-prod/04-configmap.yaml`、`06-deployment-api.yaml` 恢復 ADR-110 runtime order:
|
||||
- `OLLAMA_URL=http://192.168.0.110:11435`
|
||||
- `OLLAMA_SECONDARY_URL=http://192.168.0.110:11436`
|
||||
- `OLLAMA_FALLBACK_URL=http://192.168.0.110:11437`
|
||||
- `k8s/awoooi-prod/02-network-policy.yaml` 補 Pod → 110 `11437` egress。
|
||||
- `infra/ansible/playbooks/nginx-sync.yml` 新增舊 `conf.d/ollama-gcp-proxy.conf` 備份後移除,避免 11435/11436 duplicate server block。
|
||||
- 新增 `infra/ansible/playbooks/111-ollama-fallback.yml`,把 111 allowlist 收斂為:
|
||||
`127.0.0.1/32,192.168.0.111/32,192.168.0.188/32,192.168.0.110/32`。
|
||||
- live 111 已套用 allowlist 並重啟 LaunchAgent;live 110 已用同一份 repo template 恢復 nginx,舊 conf 已移到 `/var/backups/awoooi/nginx/`。
|
||||
|
||||
**驗證**:
|
||||
|
||||
```text
|
||||
ruby YAML load -> ok
|
||||
ansible-playbook --syntax-check:
|
||||
- nginx-sync.yml -> ok
|
||||
- 111-ollama-fallback.yml -> ok
|
||||
kubectl apply --dry-run=server -f k8s/awoooi-prod/02-network-policy.yaml -> ok
|
||||
pytest:
|
||||
DATABASE_URL=... PYTHONPATH=apps/api pytest apps/api/tests/test_ollama_endpoint_resolver.py apps/api/tests/test_ollama_failover_manager.py -q
|
||||
-> 41 passed
|
||||
ruff F/E9 targeted -> passed
|
||||
git diff --check -> passed
|
||||
kubectl kustomize k8s/awoooi-prod -> OLLAMA_URL/SECONDARY/FALLBACK resolve to 110:11435/11436/11437
|
||||
|
||||
live 110:
|
||||
nginx -T -> only sites-enabled/110-ollama-proxy.conf owns Ollama proxy
|
||||
11435 -> GCP-A upstream (currently 502 because GCP-A connection refused)
|
||||
11436 -> GCP-B upstream (currently 502 because GCP-B connection refused)
|
||||
11437 -> 111 upstream, /api/tags returns model list
|
||||
|
||||
live 111:
|
||||
com.momo.ollama111-allow-proxy launched with 192.168.0.110/32 allowed
|
||||
110 -> 192.168.0.111:11434 /api/tags returns model list
|
||||
```
|
||||
|
||||
**注意 / 下一步**:
|
||||
|
||||
- GCP-A/GCP-B upstream 仍是真正紅燈;本次先恢復「不跳過 111」的容災鏈。
|
||||
- ArgoCD 會把手動 `kubectl set env` 回滾到 Git 的舊 manifest;必須等本 commit 推到 Gitea main 後,CD/GitOps 才會讓 production API 正式吃到 `110:11435/11436/11437`。
|
||||
- 110 Ansible 實際執行仍卡在 `Incorrect sudo password`,本次 live 110 用 Docker privileged/nsenter 套用;下一階段需收斂 110/188 的 Ansible become 憑證或改成正式 rootless 管理路徑。
|
||||
|
||||
**目前整體進度**:
|
||||
|
||||
- AwoooP 告警可觀測鏈:約 99.2%。
|
||||
- 低風險自動修復閉環:約 95.5%。
|
||||
- 前端 AI 自動化管理介面同步:約 96.4%。
|
||||
- Telegram detail/history 可解釋性:約 95.5%。
|
||||
- Callback evidence / DB 回放性:約 95.6%。
|
||||
- MCP / 自建 MCP 使用可視性:約 88%。
|
||||
- Sentry / SigNoz source correlation 可視性:約 88%。
|
||||
- Ansible / PlayBook 決策可視性:約 84.5%。
|
||||
- KM owner-review / completion 可治理鏈:約 84%。
|
||||
- AI Provider lane 健康度:約 78%(111 fallback 已恢復;GCP-A/GCP-B upstream 仍待修)。
|
||||
- 完整 AI 自動化管理產品化:約 93.4%。
|
||||
|
||||
---
|
||||
|
||||
## 2026-05-25|T171 Runs list 顯示 Callback Snapshot Capture 摘要
|
||||
|
||||
**背景**:
|
||||
|
||||
Reference in New Issue
Block a user