feat(ollama): ADR-110 GCP 三層容災架構(GCP-A → GCP-B → Local → Gemini)
Some checks failed
Code Review / ai-code-review (push) Successful in 50s
CD Pipeline / tests (push) Failing after 1m14s
CD Pipeline / build-and-deploy (push) Has been skipped
CD Pipeline / post-deploy-checks (push) Has been skipped

## 變更摘要
- Primary: http://34.143.170.20:11434 (GCP-A SSD, 9x 載速 + 2x 推理)
- Secondary: http://34.21.145.224:11434 (GCP-B SSD)
- Fallback: http://192.168.0.111:11434 (M1 Pro Local HDD,最後防線)
- 廢止 ADR-105「111 唯一鐵律」,新建 ADR-110

## 核心改動
- config.py: 新增 OLLAMA_SECONDARY_URL;validator 加 GCP IP 白名單(34.143.170.20, 34.21.145.224)
- ollama_failover_manager.py: 三層 Ollama 決策矩陣;並行健康檢查三台;health_111 → health_gcp_a
- ollama_health_monitor.py: host label 萃取改為通用版(支援 GCP 公網 IP)
- failover_alerter.py: 故障/恢復主機動態顯示,不再硬編碼「Ollama 111 (GPU)」
- ollama_auto_recovery.py: notify_recovery 改為 ollama_gcp_a;recovered_host 動態
- k8s/awoooi-prod: configmap + deployment + network-policy 同步更新(egress 加 GCP /32)
- 服務層: 10 個服務檔案硬編碼 192.168.0.111 改為讀 settings.OLLAMA_URL
- 測試: URL 常數更新,新增三層容災場景,GCP IP 白名單驗證測試

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-05-03 22:49:23 +08:00
parent e45b055e0e
commit b1ef05fa8c
28 changed files with 816 additions and 355 deletions

View File

@@ -1,5 +1,6 @@
# apps/api/tests/test_model_version_probe.py
# 2026-04-27 P3.2.1 by Claude
# 2026-05-03 ogt: ADR-110 GCP 三層容災,更新 probe URL 為 GCP-A Primary
"""
model_version_probe 單元測試
==============================
@@ -60,8 +61,8 @@ def _tags_body(models: list[dict]) -> dict:
class TestProbeOllamaVersion:
@pytest.mark.asyncio
async def test_success_111_provider(self):
"""111 URL → provider='ollama', digest 和 version 正確解析"""
async def test_success_gcp_a_provider(self):
"""GCP-A URL → provider='ollama', digest 和 version 正確解析ADR-110"""
model_entry = {
"name": "qwen2.5:7b-instruct",
"modified_at": "2026-04-01T00:00:00Z",
@@ -79,7 +80,7 @@ class TestProbeOllamaVersion:
with patch("httpx.AsyncClient", return_value=mock_client):
info = await probe_ollama_version(
"http://192.168.0.111:11434", "qwen2.5:7b-instruct"
"http://34.143.170.20:11434", "qwen2.5:7b-instruct"
)
assert info.provider == "ollama"
@@ -123,7 +124,7 @@ class TestProbeOllamaVersion:
with patch("httpx.AsyncClient", return_value=mock_client):
with pytest.raises(ValueError, match="not found"):
await probe_ollama_version(
"http://192.168.0.111:11434", "qwen2.5:7b-instruct"
"http://34.143.170.20:11434", "qwen2.5:7b-instruct"
)
@pytest.mark.asyncio
@@ -139,7 +140,7 @@ class TestProbeOllamaVersion:
with patch("httpx.AsyncClient", return_value=mock_client):
with pytest.raises(httpx.HTTPStatusError):
await probe_ollama_version(
"http://192.168.0.111:11434", "qwen2.5:7b-instruct"
"http://34.143.170.20:11434", "qwen2.5:7b-instruct"
)
@pytest.mark.asyncio
@@ -153,7 +154,7 @@ class TestProbeOllamaVersion:
with patch("httpx.AsyncClient", return_value=mock_client):
with pytest.raises(httpx.TimeoutException):
await probe_ollama_version(
"http://192.168.0.111:11434", "qwen2.5:7b-instruct"
"http://34.143.170.20:11434", "qwen2.5:7b-instruct"
)
@@ -345,7 +346,7 @@ class TestProbeAllProviders:
patch("src.services.model_version_probe.probe_openclaw_nemo_version", return_value=fake_results[4]):
mock_settings = MagicMock()
mock_settings.OLLAMA_URL = "http://192.168.0.111:11434"
mock_settings.OLLAMA_URL = "http://34.143.170.20:11434" # GCP-AADR-110
mock_settings.OLLAMA_FALLBACK_URL = "http://192.168.0.188:11434"
mock_settings.OLLAMA_HEALTH_CHECK_MODEL = "qwen2.5:7b-instruct"
@@ -377,7 +378,7 @@ class TestProbeAllProviders:
)):
mock_settings = MagicMock()
mock_settings.OLLAMA_URL = "http://192.168.0.111:11434"
mock_settings.OLLAMA_URL = "http://34.143.170.20:11434" # GCP-AADR-110
mock_settings.OLLAMA_FALLBACK_URL = "http://192.168.0.188:11434"
mock_settings.OLLAMA_HEALTH_CHECK_MODEL = "qwen2.5:7b-instruct"