fix(ai): bind PixelRAG VLM worker to selected route

2026-07-10 00:23:05 +08:00
parent bed3a8a8ee
commit 9cf8b55b72
5 changed files with 150 additions and 47 deletions
--- a/TODO_NEXT_STEPS.txt
+++ b/TODO_NEXT_STEPS.txt
@@ -9,7 +9,7 @@
    python scripts/ops/check_production_version_truth.py

 目前最新版本仍以 production `https://mo.wooo.work/health` readback 為準。
-本輪 source target 為 `V10.753`；部署完成前不得宣稱正式環境已是 `V10.753`。
+本輪 source target 為 `V10.754`；部署完成前不得宣稱正式環境已是 `V10.754`。
 舊 iCloud checkout 不是 Gitea dev worktree，不得拿來當最新版本真相。

 ================================================================================
--- a/config.py
+++ b/config.py
@@ -402,7 +402,7 @@ YOUTUBE_API_KEY = os.getenv('YOUTUBE_API_KEY', '')
 # ==========================================
 # 系統版本與路徑
 # ==========================================
-SYSTEM_VERSION = "V10.753"
+SYSTEM_VERSION = "V10.754"
 LOG_FILE_PATH = os.path.join(BASE_DIR, 'logs/system.log')
 public_url = PUBLIC_URL  # 用於模板顯示

--- a/docs/AI_INTELLIGENCE_MODULE_SOT.md
+++ b/docs/AI_INTELLIGENCE_MODULE_SOT.md
@@ -883,6 +883,7 @@ POSTGRES_HOST=momo-db
 | 2026-07-09 | PixelRAG Ollama-first VLM replay worker 必須有 runtime monitoring | V10.751 起 `/api/ai-automation/pixelrag-vlm-replay-worker`、`scripts/ops/run_pixelrag_vlm_replay_worker.py`、`/api/ai-automation/smoke` 與 `/api/ai-automation/scheduled-health-summary` 必須輸出 controlled VLM replay worker / `pixelrag_vlm_replay_worker` family；readback 預設 dry-run，不呼叫模型、不寫 artifact，execute 模式只讀 saved tiles、呼叫 approved Ollama VLM route、驗證 JSON field confidence/evidence refs，並只寫 artifact receipt；model_error 也必須寫 failure artifact receipt，receipt 檔內需自證 `artifact_write_performed=true` 與 `receipt_path`。此 worker 明確標記 `writes_database=false`、`writes_ai_insights=false`、`writes_price_tables=false`、`secret_read=false`、`primary_human_gate_count=0`；blocked page 不得輸出商品欄位，ready VLM 結果仍需 identity matcher replay 與 PromotionGate。 |
 | 2026-07-09 | PixelRAG VLM route readiness 必須有 runtime monitoring 與 execute 前自動選模 | V10.752 起 `/api/ai-automation/pixelrag-vlm-route-readiness`、`scripts/ops/report_pixelrag_vlm_route_readiness.py`、`/api/ai-automation/smoke` 與 `/api/ai-automation/scheduled-health-summary` 必須輸出 read-only approved Ollama route/model readiness / `pixelrag_vlm_route_readiness` family；readback 只打 `/api/tags`，不呼叫 `/api/generate`，輸出 reachable host、configured model available count、candidate model、candidate host、model_route_ready 與 next machine action。`run_pixelrag_vlm_replay_worker.py --execute` 預設必須使用此 readback 自動選擇已安裝候選模型；完全沒有候選時寫 `model_route_not_ready` artifact receipt，不得盲打 missing model。 |
 | 2026-07-10 | PixelRAG VLM execute preflight 必須驗證 generate route | V10.753 起 `/api/ai-automation/pixelrag-vlm-route-readiness?probe_generate=true` 與 `scripts/ops/report_pixelrag_vlm_route_readiness.py --probe-generate` 可對已安裝候選模型執行極小 `/api/generate` preflight；smoke 預設仍不呼叫模型。`run_pixelrag_vlm_replay_worker.py --execute` 預設先執行 generate preflight，若 GCP-A direct / 110 proxy timeout、GCP-B candidate 雖安裝但 generate 不健康，worker 必須在送入 screenshot tiles 前寫 `model_route_not_ready` artifact receipt，輸出 `tag_model_route_ready`、`generate_route_ready`、`route_model_call_performed` 與 `tile_model_call_performed=false`，下一步固定為 `repair_ollama_vlm_generate_runtime_or_proxy_timeout`；不得把 tags 可見誤當 VLM 可執行。 |
+| 2026-07-10 | PixelRAG VLM tile generate 必須綁定 preflight 選出的 host / exact model | V10.754 起 `run_pixelrag_vlm_replay_worker.py --execute` 若 route readiness 已輸出 `candidate_host`，tile VLM generate 必須使用該 approved host 與 exact `candidate_model` 直呼 `/api/generate`，不得重新進入全域 Ollama resolver、不得因 111 fallback 規則把 VLM 模型降級成文字模型、不得再出現 preflight 選 111 但 tile generate tried GCP-B/GCP-A/111 的 route drift。receipt 必須輸出 `route_candidate_host`，模型錯誤時下一步為 `repair_ollama_vlm_generate_runtime_or_proxy_timeout`，仍然不寫 DB、不寫正式價格表。 |
 | 2026-06-29 | PChome DB apply 授權 lane 必須先通過 no-write guard / decision preflight / decision closeout / issuer gate / signing-decision preflight / signing-decision closeout / signing-issuer guard | V10.725 的 PChome mapping backlog auto-policy 已新增 `/api/ai/pchome-growth/mapping-backlog/auto-policy-db-apply-authorization-lane-guard`、`/api/ai/pchome-growth/mapping-backlog/auto-policy-db-apply-authorization-decision-preflight`、`/api/ai/pchome-growth/mapping-backlog/auto-policy-db-apply-authorization-decision-closeout`、`/api/ai/pchome-growth/mapping-backlog/auto-policy-db-apply-authorization-issuer-gate`、`/api/ai/pchome-growth/mapping-backlog/auto-policy-db-apply-authorization-signing-decision-preflight`、`/api/ai/pchome-growth/mapping-backlog/auto-policy-db-apply-authorization-signing-decision-closeout` 與 `/api/ai/pchome-growth/mapping-backlog/auto-policy-db-apply-authorization-signing-issuer-guard`；這些 endpoint 只驗證 final exact request package、same-run production truth requirement、secret rejection、rollback boundary、lane entry requirements、decision input requirements、rejection policy、post-apply verifier、future authorization decision package、final nonsecret authorization envelope、signing decision preflight inputs、unsigned signing decision package 與 signable request boundary，不讀 secret、不執行 shell/SQL、不寫 DB，也不簽發 database apply authorization。 |
 | 2026-06-29 | PChome DB apply 授權簽署發行者 lane 必須先產出 final signable request package | V10.725 的 PChome mapping backlog auto-policy 新增 `/api/ai/pchome-growth/mapping-backlog/auto-policy-db-apply-authorization-signing-issuer-closeout`；此 endpoint 只把 signing-issuer guard 的 signable request boundary 收斂成 final signable request package 與 closeout contract，確認 fresh production truth、post-apply verifier、migration hash、secret boundary 與 no-side-effect checks，不讀 secret、不簽發 authorization、不執行 shell/SQL、不寫 DB，也不代表正式 DB apply 已授權。 |
 | 2026-06-29 | PChome DB apply 授權簽署執行 lane 必須先通過 operator-held secret boundary preflight | V10.725 的 PChome mapping backlog auto-policy 新增 `/api/ai/pchome-growth/mapping-backlog/auto-policy-db-apply-authorization-signing-execution-preflight`；此 endpoint 只把 final signable request package 轉成 future signing execution preflight package、operator-held secret boundary contract、nonsecret signing inputs、command-shape preview、rollback boundary 與 abort conditions，不讀 secret、不接受 plaintext secret、不簽發 authorization、不執行 shell/SQL、不寫 DB，也不代表正式 DB apply 已授權。 |
--- a/services/pixelrag_vlm_replay_worker_service.py
+++ b/services/pixelrag_vlm_replay_worker_service.py
@@ -16,7 +16,15 @@ from datetime import datetime, timezone
 from pathlib import Path
 from typing import Any, Mapping

-from services.ollama_service import OllamaService, get_host_label, get_provider_tag
+import requests
+
+from services.ollama_service import (
+    OllamaResponse,
+    OllamaService,
+    get_host_label,
+    get_provider_tag,
+    is_approved_ollama_host,
+)
 from services.pixelrag_crawler_integration_service import (
    DEFAULT_ARTIFACT_MAX_AGE_HOURS,
    DEFAULT_ARTIFACT_ROOT,
@@ -223,6 +231,78 @@ def _validate_model_payload(
    }


+def _generate_exact_host(
+    prompt: str,
+    *,
+    host: str,
+    model: str,
+    temperature: float,
+    timeout: int,
+    options: Mapping[str, Any] | None,
+    images: list[str],
+) -> OllamaResponse:
+    """Call the route-readiness selected host without fallback or model downgrade."""
+    clean_host = str(host or "").rstrip("/")
+    if not is_approved_ollama_host(clean_host):
+        return OllamaResponse(
+            success=False,
+            content="",
+            model=model,
+            error=f"unapproved_pixelrag_vlm_candidate_host: {clean_host}",
+            host=clean_host or "unknown",
+        )
+    payload: dict[str, Any] = {
+        "model": model,
+        "prompt": prompt,
+        "stream": False,
+        "options": {"temperature": temperature},
+    }
+    if options:
+        payload["options"].update(dict(options))
+    if images:
+        payload["images"] = images
+    try:
+        response = requests.post(
+            f"{clean_host}/api/generate",
+            json=payload,
+            timeout=max(1, int(timeout or DEFAULT_TIMEOUT_SECONDS)),
+        )
+        if response.status_code != 200:
+            return OllamaResponse(
+                success=False,
+                content="",
+                model=model,
+                error=f"HTTP {response.status_code}: {response.text[:RAW_EXCERPT_LIMIT]}",
+                host=clean_host,
+            )
+        data = response.json()
+        return OllamaResponse(
+            success=True,
+            content=data.get("response", ""),
+            model=model,
+            total_duration=(data.get("total_duration", 0) or 0) / 1e9,
+            host=clean_host,
+            input_tokens=int(data.get("prompt_eval_count", 0) or 0),
+            output_tokens=int(data.get("eval_count", 0) or 0),
+        )
+    except requests.Timeout:
+        return OllamaResponse(
+            success=False,
+            content="",
+            model=model,
+            error=f"timeout ({max(1, int(timeout or DEFAULT_TIMEOUT_SECONDS))}s)",
+            host=clean_host,
+        )
+    except Exception as exc:
+        return OllamaResponse(
+            success=False,
+            content="",
+            model=model,
+            error=f"{type(exc).__name__}: {str(exc)[:RAW_EXCERPT_LIMIT]}",
+            host=clean_host,
+        )
+
+
 def _write_replay_receipt(
    *,
    output_root: Path,
@@ -324,6 +404,7 @@ def _execute_item(
    root: Path,
    output_root: Path,
    model: str,
+    route_host: str | None,
    timeout: int,
    tile_limit: int,
    write_receipt: bool,
@@ -335,6 +416,7 @@ def _execute_item(
        "source_receipt_path": item.get("source_receipt_path"),
        "worker_status": "executing",
        "model": model,
+        "route_candidate_host": str(route_host or ""),
        "tile_evidence": tile_evidence,
        "tile_image_count": len(images),
        "model_call_performed": bool(images),
@@ -348,14 +430,27 @@ def _execute_item(
        })
        return base

-    response = OllamaService(model=model).generate(
-        _prompt_for_item(item),
-        model=model,
-        temperature=0.1,
-        timeout=max(10, int(timeout or DEFAULT_TIMEOUT_SECONDS)),
-        options={"num_predict": 700, "num_ctx": 4096},
-        images=images,
-    )
+    prompt = _prompt_for_item(item)
+    options = {"num_predict": 700, "num_ctx": 4096}
+    if route_host:
+        response = _generate_exact_host(
+            prompt,
+            host=route_host,
+            model=model,
+            temperature=0.1,
+            timeout=max(10, int(timeout or DEFAULT_TIMEOUT_SECONDS)),
+            options=options,
+            images=images,
+        )
+    else:
+        response = OllamaService(model=model).generate(
+            prompt,
+            model=model,
+            temperature=0.1,
+            timeout=max(10, int(timeout or DEFAULT_TIMEOUT_SECONDS)),
+            options=options,
+            images=images,
+        )
    base.update({
        "host": response.host,
        "host_label": get_host_label(response.host or ""),
@@ -369,7 +464,11 @@ def _execute_item(
        base.update({
            "worker_status": "model_error",
            "model_error": str(response.error or "")[:RAW_EXCERPT_LIMIT],
-            "next_machine_action": "repair_ollama_vlm_runtime_or_model_route",
+            "next_machine_action": (
+                "repair_ollama_vlm_generate_runtime_or_proxy_timeout"
+                if route_host
+                else "repair_ollama_vlm_runtime_or_model_route"
+            ),
        })
        if write_receipt:
            base["receipt_path"] = _write_replay_receipt(
@@ -454,6 +553,7 @@ def run_pixelrag_ollama_vlm_replay_worker(
    item_limit = max(1, min(int(limit or DEFAULT_LIMIT), 250))
    tiles = max(1, min(int(tile_limit or DEFAULT_TILE_LIMIT), 12))
    selected_model = str(model or DEFAULT_MODEL)
+    selected_route_host = ""
    selected_timeout = max(10, int(timeout or DEFAULT_TIMEOUT_SECONDS))
    readiness_timeout = max(1, min(int(route_readiness_timeout or 3), 20))
    generate_probe_timeout = max(
@@ -489,6 +589,7 @@ def run_pixelrag_ollama_vlm_replay_worker(
        model_route_ready = bool(route_summary.get("model_route_ready"))
        if candidate_model:
            selected_model = candidate_model
+            selected_route_host = str(route_summary.get("candidate_host") or "").strip()

    worker_items: list[dict[str, Any]] = []
    for item in replay_items:
@@ -511,6 +612,7 @@ def run_pixelrag_ollama_vlm_replay_worker(
            root=root,
            output_root=output,
            model=selected_model,
+            route_host=selected_route_host,
            timeout=selected_timeout,
            tile_limit=tiles,
            write_receipt=write_receipt,
@@ -607,6 +709,7 @@ def run_pixelrag_ollama_vlm_replay_worker(
        "tile_limit": tiles,
        "model": selected_model,
        "configured_model": str(model or DEFAULT_MODEL),
+        "route_candidate_host": selected_route_host,
        "timeout_seconds": selected_timeout,
        "execute": bool(execute),
        "write_receipt": bool(write_receipt),
--- a/tests/test_pixelrag_vlm_replay_worker_service.py
+++ b/tests/test_pixelrag_vlm_replay_worker_service.py
@@ -204,44 +204,41 @@ def test_pixelrag_vlm_replay_worker_auto_selects_installed_candidate(tmp_path, m
        },
    )

-    class FakeOllama:
-        def __init__(self, model):
-            self.model = model
-
-        def generate(self, *args, **kwargs):
-            assert kwargs["model"] == "gemma3:4b"
-            return SimpleNamespace(
-                success=True,
-                content=json.dumps({
-                    "blocked_page_detected": False,
-                    "fields": {
-                        "title": {
-                            "value": "防曬乳 SPF50",
-                            "confidence": 0.92,
-                            "evidence_refs": ["tile:1"],
-                        },
-                        "price": {
-                            "value": "399",
-                            "confidence": 0.88,
-                            "evidence_refs": ["tile:1"],
-                        },
+    def fake_generate_exact_host(prompt, **kwargs):
+        assert kwargs["host"] == "http://34.21.145.224:11434"
+        assert kwargs["model"] == "gemma3:4b"
+        return SimpleNamespace(
+            success=True,
+            content=json.dumps({
+                "blocked_page_detected": False,
+                "fields": {
+                    "title": {
+                        "value": "防曬乳 SPF50",
+                        "confidence": 0.92,
+                        "evidence_refs": ["tile:1"],
                    },
-                    "quality": {
-                        "overall_confidence": 0.90,
-                        "missing_required_fields": [],
-                        "requires_identity_matcher_replay": True,
-                        "requires_promotion_gate": True,
+                    "price": {
+                        "value": "399",
+                        "confidence": 0.88,
+                        "evidence_refs": ["tile:1"],
                    },
-                }),
-                model="gemma3:4b",
-                error=None,
-                total_duration=1.0,
-                host="http://34.21.145.224:11434",
-                input_tokens=10,
-                output_tokens=50,
-            )
+                },
+                "quality": {
+                    "overall_confidence": 0.90,
+                    "missing_required_fields": [],
+                    "requires_identity_matcher_replay": True,
+                    "requires_promotion_gate": True,
+                },
+            }),
+            model="gemma3:4b",
+            error=None,
+            total_duration=1.0,
+            host="http://34.21.145.224:11434",
+            input_tokens=10,
+            output_tokens=50,
+        )

-    monkeypatch.setattr(service, "OllamaService", FakeOllama)
+    monkeypatch.setattr(service, "_generate_exact_host", fake_generate_exact_host)
    payload = service.run_pixelrag_ollama_vlm_replay_worker(
        artifact_root=tmp_path,
        platform="shopee_tw",
@@ -253,7 +250,9 @@ def test_pixelrag_vlm_replay_worker_auto_selects_installed_candidate(tmp_path, m

    assert payload["model"] == "gemma3:4b"
    assert payload["configured_model"] == "minicpm-v:latest"
+    assert payload["route_candidate_host"] == "http://34.21.145.224:11434"
    assert payload["route_readiness"]["summary"]["candidate_model"] == "gemma3:4b"
+    assert payload["worker_items"][0]["route_candidate_host"] == "http://34.21.145.224:11434"
    assert payload["summary"]["executed_ok_count"] == 1