diff --git a/.gitignore b/.gitignore index b481110..8a8d457 100644 --- a/.gitignore +++ b/.gitignore @@ -90,6 +90,7 @@ data/excel_exports/ # 本機 QA / 頁面快取 data/*_cache/ data/ai_automation_smoke_history.jsonl +data/ai_automation/sitewide_visual_qa_latest.json # 上傳檔案 web/static/uploads/ diff --git a/docs/AI_INTELLIGENCE_MODULE_SOT.md b/docs/AI_INTELLIGENCE_MODULE_SOT.md index c2047e6..ac44d71 100644 --- a/docs/AI_INTELLIGENCE_MODULE_SOT.md +++ b/docs/AI_INTELLIGENCE_MODULE_SOT.md @@ -111,6 +111,7 @@ - 2026-07-03 起高可見 AI surface HTML readback 必須納入 runtime AI smoke:`/api/ai-automation/surface-html-readback` 會 read-only 檢查 10 個 AI observability template contract,`/api/ai-automation/smoke` 需包含 `AI surface HTML readback` 檢查,`/api/ai-automation/scheduled-health-summary` 需輸出 `ai_surface_html_readback` family;此檢查不得寫 DB、不得送通知、不得要求 browser,失敗時下一個機器動作是 `repair_ai_surface_html_contract`。 - 2026-07-03 起整站 UI/UX 專業化必須由 Sitewide UI/UX Agent 管理:`/api/ai-automation/sitewide-ui-ux-agent` 會掃描真正頁面模板、排除 partial/components/base layouts,輸出高優先頁、compact guardrail、raw engineering issue 與 next machine action;`/api/ai-automation/sitewide-ui-ux-repair-package` 會產生 no-write controlled repair package。當前 Agent 判定為 55 個頁面模板、31 個高優先模板、29 個已具 compact guardrail、0 個待專業化、0 個 raw engineering leak,Sitewide Agent 狀態為 `ok`,repair package 為 `no_op`。四批 controlled UI/UX repair 已完成 AI observability、AI 建議、PPT/CI/CD/業績/比價/成長/供貨風險高優先 surface;下一個機器動作為 `keep_sitewide_ui_ux_agent_monitoring`,後續只接受 Agent 新增 issue 或更高階 visual/mobile QA 的證據驅動修復。 - 2026-07-03 起 Sitewide UI/UX Agent 歸零狀態必須納入 AI automation runtime monitoring:`/api/ai-automation/smoke` 需包含 `Sitewide UI/UX Agent readback` 檢查,要求 `issue_surface_count=0`、`repair_package_status=no_op`、`repair_action_count=0`;`/api/ai-automation/scheduled-health-summary` 需輸出 `sitewide_ui_ux_agent` family;`/metrics` 需以 `momo_ai_automation_scheduled_health_family_status{family="sitewide_ui_ux_agent",...}` 暴露 scheduled family。此監控不得寫 DB、不得送通知、不得要求人工審核。 +- 2026-07-03 起整站 desktop/tablet/mobile visual QA 必須進入 AI automation runtime monitoring:`scripts/check_responsive_overflow.js --artifact-output data/ai_automation/sitewide_visual_qa_latest.json` 會輸出機器可讀 artifact;`/api/ai-automation/sitewide-visual-qa-readback` 只讀該 artifact 並回報 route_count、viewport_count、failed_count、overflow_issue_count、stale 與 next_machine_action;`/api/ai-automation/smoke` 需包含 `Sitewide visual QA readback`,`/api/ai-automation/scheduled-health-summary` 需輸出 `sitewide_visual_qa` family,`/metrics` 需暴露 `momo_ai_automation_scheduled_health_family_status{family="sitewide_visual_qa",...}`。readback 不啟動 browser、不寫 DB、不送通知;artifact 生成才需要 Playwright/瀏覽器。 - 2026-07-02 起 `/ai_intelligence` 商品明細與單品作戰詳情的四格價格證據必須可測:PChome 價格、MOMO 參考價、差距、可信度需以 `data-evidence` 固定,並以 `aria-label="價格證據"` 對應可掃描區塊;候選待確認或缺資料只能顯示「候選待確認 / 待補」,不得捏造價格或讓使用者打開 raw payload 才知道判斷依據。 - 2026-07-02 起 `/ai_intelligence` 必須是密集 AI 工作台,不得退回大段文字說明頁:首屏與明細可見內容只保留短狀態、數字、四格證據與下一步按鈕;KPI note、benchmark detail、alert 副句、策略說明、decision copy、來源長句與單品 reason list 不得佔用第一層視覺。`tests/test_ai_intelligence_text_density_guardrails.py` 必須鎖住 `data-density-guardrail="compact-ai-workbench"`、短任務文案、detail meta 與 hidden explanatory copy。 - 2026-07-02 起 `/observability/overview` 也必須採密集 AI 觀測工作台:首屏以 `data-density-guardrail="compact-observability-workbench"`、`AI 觀測 / 風險優先 / 下一步` 與 golden signals 先呈現狀態、數字與操作入口;hero lede、signal note、route desc、host meta 與資料來源長句不得佔用第一層視覺。`tests/test_observability_text_density_guardrails.py` 必須鎖住 compact marker 與 hidden explanatory copy。 diff --git a/docs/guides/external_professional_benchmark.md b/docs/guides/external_professional_benchmark.md index 271d606..7ef8a91 100644 --- a/docs/guides/external_professional_benchmark.md +++ b/docs/guides/external_professional_benchmark.md @@ -84,6 +84,7 @@ Baymard 的商品頁與比較 UX 研究強調:使用者需要清楚的 product - Route HTML readback rollout: high-visibility AI observability surfaces 已新增 Flask route-level rendered HTML guard;`tests/test_admin_observability_routes.py::test_high_visibility_ai_surfaces_route_html_readback_keeps_compact_contract` 鎖住 10 個 route 的 compact density marker 與 non-raw engineering wording。 - Runtime HTML readback rollout: `/api/ai-automation/surface-html-readback` 與 AI smoke 已接入 high-visibility AI surface contract readback;`tests/test_ai_automation_smoke_service.py` 鎖住 10 個 surface、退化偵測、smoke check 與 scheduled health family。 - Sitewide UI/UX Agent rollout: `/api/ai-automation/sitewide-ui-ux-agent` 掃描真正頁面模板並產生整站專業化 inventory;`/api/ai-automation/sitewide-ui-ux-repair-package` 會輸出 no-write controlled repair package,把主流產品頁原則轉成可排序、可驗證的修復項。四批 controlled repair 已把 AI 建議、PPT 預覽、服務更新監控、當日業績、商品/EDM 舊入口、成長報表、PChome 商品監控、比價決策台、業績作戰分析與供貨風險頁群納入 compact workbench guardrails,Agent 基線收斂到 29 個 compact guardrail / 0 個待專業化 surface。 +- Sitewide visual QA rollout: `scripts/check_responsive_overflow.js --artifact-output data/ai_automation/sitewide_visual_qa_latest.json` 會把 desktop / tablet / mobile overflow QA 輸出為 artifact;`/api/ai-automation/sitewide-visual-qa-readback`、`Sitewide visual QA readback` 與 `sitewide_visual_qa` scheduled family 會把最新視覺 QA 狀態納入 AI automation monitoring。 ## 下一步 TODO 候選 @@ -103,6 +104,7 @@ Baymard 的商品頁與比較 UX 研究強調:使用者需要清楚的 product - 已完成: `/observability/rag_queries` 首屏以「知識命中 / 省模閉環 / 回饋學習」短標籤與四個核心數字呈現 AI 知識召回狀態。 - 已完成: `/observability/promotion_review` 首屏以「AI 例外 / 去重守門 / 防污染」短標籤與四個核心數字呈現 AI 晉升例外狀態。 - 已完成: `/observability/ppt_audit_history` 首屏以「預覽就緒 / 視覺審核 / 修復閉環」短標籤呈現 AI 視覺 QA 產線狀態。 + - 已完成: Sitewide visual QA artifact readback 會把 desktop / tablet / mobile overflow 結果接入 AI smoke、scheduled health summary 與 `/metrics`。 ## 參考來源 diff --git a/docs/guides/pchome_ai_automation_priority_backlog.md b/docs/guides/pchome_ai_automation_priority_backlog.md index 98d577f..68853d5 100644 --- a/docs/guides/pchome_ai_automation_priority_backlog.md +++ b/docs/guides/pchome_ai_automation_priority_backlog.md @@ -179,9 +179,14 @@ - 此檢查不寫 DB、不發通知、不要求 browser;失敗時 next machine action 是 `repair_ai_surface_html_contract`。 - 整站 UI/UX Agent inventory 已完成: - `/api/ai-automation/sitewide-ui-ux-agent` 會掃描真正頁面模板,排除 partial/components/base layouts。 - - 當前基線: 55 個頁面模板、31 個高優先模板、14 個 compact guardrail、15 個待專業化、0 個 raw engineering leak。 + - 當前基線: 55 個頁面模板、31 個高優先模板、29 個 compact guardrail、0 個待專業化 surface、0 個 raw engineering leak。 - `/api/ai-automation/sitewide-ui-ux-repair-package` 會產生 no-write controlled repair package,優先修復高曝光頁,且不寫 DB、不執行 shell、不讀 secrets。 - - 第一批 controlled UI/UX repair 已套用到 `ai_history` 與 `ai_recommend`,整站 Agent 已回讀 issue count 從 17 降為 15。 + - 四批 controlled UI/UX repair 已完成 AI observability、AI 建議、PPT/CI/CD/業績/比價/成長/供貨風險高優先 surface,整站 repair package 目前為 `no_op`。 +- 整站 visual/mobile QA artifact readback 已完成: + - `scripts/check_responsive_overflow.js --artifact-output data/ai_automation/sitewide_visual_qa_latest.json` 會把 desktop / tablet / mobile overflow QA 輸出為 machine-readable artifact。 + - `/api/ai-automation/sitewide-visual-qa-readback` 只讀 artifact,回報 route_count、viewport_count、failed_count、overflow_issue_count、stale 與 next_machine_action。 + - `/api/ai-automation/smoke` 已加入 `Sitewide visual QA readback`,scheduled health summary 已加入 `sitewide_visual_qa` family,`/metrics` 會輸出對應 Prometheus family status。 + - readback 不寫 DB、不送通知、不啟動 browser;artifact 生成才需要 Playwright/瀏覽器,避免把 visual QA 變成產品 API 的重型副作用。 已完成 / 下一步,必須照順序: @@ -196,6 +201,8 @@ - 已完成: 高可見 AI surface readback 已接入 runtime smoke 與 scheduled health summary。 4. 啟動整站 UI/UX Agent,將所有頁面依主流專業 guardrails 排序並產生 controlled repair package。 - 已完成: Sitewide UI/UX Agent inventory 與 repair package API。 +5. 將 desktop / tablet / mobile visual QA 變成 AI automation artifact 與 runtime readback。 + - 已完成: Sitewide visual QA artifact schema、readback API、AI smoke、scheduled family 與 metrics family。 完成標準: diff --git a/docs/memory/current_execution_queue_20260524.md b/docs/memory/current_execution_queue_20260524.md index ad0fb4c..9c7d0d7 100644 --- a/docs/memory/current_execution_queue_20260524.md +++ b/docs/memory/current_execution_queue_20260524.md @@ -174,6 +174,7 @@ - 表格與圖表只允許在局部容器橫向滾動,不可造成整頁無限延伸。 - 2026-07-03 起以 Sitewide UI/UX Agent 排序全站專業化工作,不再靠人工逐頁猜測;目前 55 個頁面模板 / 31 個高優先頁中,compact guardrail 已由 14 提升到 29,待專業化 surface 由 15 降到 0。第四批已完成 vendor stockout 剩餘五頁,repair package 進入 `no_op`,下一步轉為 production monitoring 與 desktop/mobile visual QA。 - 2026-07-03 起 Sitewide UI/UX Agent 已接入 AI automation smoke、scheduled health summary 與 `/metrics` family;之後若整站 UI/UX 退化,主線入口是 `Sitewide UI/UX Agent readback` / `sitewide_ui_ux_agent` family,而不是人工逐頁盤點。 +- 2026-07-03 起 desktop/tablet/mobile visual QA artifact 也接入 AI automation:`scripts/check_responsive_overflow.js --artifact-output data/ai_automation/sitewide_visual_qa_latest.json` 產生 Playwright overflow artifact,`/api/ai-automation/sitewide-visual-qa-readback`、`Sitewide visual QA readback` 與 `sitewide_visual_qa` scheduled family 會讀取最新結果。後續若手機/桌面 overflow 或 artifact stale,主線入口是重新跑 visual QA artifact 並修復 failed routes,不再靠人工截圖逐頁判斷。 ## 8. 效能與可觀測性 diff --git a/routes/README.md b/routes/README.md index d3e4494..b74c760 100644 --- a/routes/README.md +++ b/routes/README.md @@ -14,7 +14,7 @@ |------|------|----------| | `dashboard_routes.py` | 商品看板首頁 | `/` | | `sales_routes.py` | 業績分析與 ABC 明細 | `/sales_analysis`, `/growth_analysis`, `/abc_analysis/detail`, `/api/sales_analysis/*` | -| `system_public_routes.py` | 無 prefix 公開系統頁與監控 | `/health`, `/metrics`, `/ai_automation_smoke`, `/api/ai-automation/smoke*`, `/api/ai-automation/scheduled-health-summary`, `/api/ai-automation/surface-html-readback`, `/api/ai-automation/surface-html-repair-package`, `/api/ai-automation/sitewide-ui-ux-agent`, `/api/ai-automation/sitewide-ui-ux-repair-package`, `/settings`, `/system_settings`, `/logs`, `/api/logs`, `/api/backup` | +| `system_public_routes.py` | 無 prefix 公開系統頁與監控 | `/health`, `/metrics`, `/ai_automation_smoke`, `/api/ai-automation/smoke*`, `/api/ai-automation/scheduled-health-summary`, `/api/ai-automation/surface-html-readback`, `/api/ai-automation/surface-html-repair-package`, `/api/ai-automation/sitewide-ui-ux-agent`, `/api/ai-automation/sitewide-ui-ux-repair-package`, `/api/ai-automation/sitewide-visual-qa-readback`, `/settings`, `/system_settings`, `/logs`, `/api/logs`, `/api/backup` | | `system_routes.py` | 內部系統維護 API | `/api/system/*` | | `edm_routes.py` | EDM 與節慶儀表板 | `/edm`, `/festival` | | `monthly_routes.py` | 月結分析 | `/monthly_summary_analysis`, `/api/monthly_summary_data` | diff --git a/routes/system_public_routes.py b/routes/system_public_routes.py index b07a79f..eda04d4 100644 --- a/routes/system_public_routes.py +++ b/routes/system_public_routes.py @@ -624,6 +624,16 @@ def ai_automation_sitewide_ui_ux_repair_package_api(): return jsonify(build_sitewide_ui_ux_repair_package(limit=max(1, min(limit or 12, 50)))) +@system_public_bp.route('/api/ai-automation/sitewide-visual-qa-readback') +@login_required +def ai_automation_sitewide_visual_qa_readback_api(): + """Read-only sitewide desktop/tablet/mobile visual QA artifact readback.""" + from services.ai_surface_html_readback_service import build_sitewide_visual_qa_readback + + max_age_hours = request.args.get('max_age_hours', 26, type=int) + return jsonify(build_sitewide_visual_qa_readback(max_age_hours=max(1, min(max_age_hours or 26, 168)))) + + @system_public_bp.route('/api/ai-automation/smoke/history/export') @login_required def ai_automation_smoke_history_export(): diff --git a/scripts/check_responsive_overflow.js b/scripts/check_responsive_overflow.js index d84e3da..1408a86 100755 --- a/scripts/check_responsive_overflow.js +++ b/scripts/check_responsive_overflow.js @@ -110,6 +110,7 @@ function parseArgs(argv) { maxOverflow: 1, screenshotDir: '', screenshotAll: false, + artifactOutput: '', json: false, }; @@ -139,6 +140,8 @@ function parseArgs(argv) { options.screenshotDir = argv[++i]; } else if (arg === '--screenshot-all') { options.screenshotAll = true; + } else if (arg === '--artifact-output') { + options.artifactOutput = argv[++i]; } else if (arg === '--json') { options.json = true; } else if (arg === '--help' || arg === '-h') { @@ -173,6 +176,7 @@ Options: --max-overflow PX Allowed body overflow, default 1 --screenshot-dir DIR Save failure screenshots --screenshot-all Save screenshots for passing routes too + --artifact-output PATH Write machine-readable sitewide visual QA artifact --json Print JSON summary `); } @@ -315,6 +319,39 @@ function printResult(result) { } } +function buildArtifact(options, results) { + const failed = results.filter((result) => !result.passed); + const routes = Array.from(new Set(results.map((result) => result.route))).sort(); + const viewports = Array.from(new Set(results.map((result) => result.viewport))).sort(); + return { + policy: 'sitewide_responsive_overflow_guard_v1', + generated_at: new Date().toISOString(), + base_url: options.baseUrl, + max_overflow_px: options.maxOverflow, + route_count: routes.length, + viewport_count: viewports.length, + summary: { + result_count: results.length, + pass_count: results.length - failed.length, + failed_count: failed.length, + overflow_issue_count: failed.filter((result) => String(result.error || '').toLowerCase().includes('overflow')).length, + visual_offender_count: failed.filter((result) => String(result.error || '').toLowerCase().includes('offender')).length, + }, + routes, + viewports, + results, + }; +} + +function writeArtifact(options, artifact) { + if (!options.artifactOutput) { + return; + } + const outputPath = path.resolve(options.artifactOutput); + fs.mkdirSync(path.dirname(outputPath), { recursive: true }); + fs.writeFileSync(outputPath, `${JSON.stringify(artifact, null, 2)}\n`, 'utf-8'); +} + async function main() { const options = parseArgs(process.argv.slice(2)); const { chromium } = requirePlaywright(); @@ -395,6 +432,9 @@ async function main() { await closeWithTimeout(browser); } + const artifact = buildArtifact(options, results); + writeArtifact(options, artifact); + if (options.json) { console.log(JSON.stringify(results, null, 2)); } diff --git a/services/ai_automation_smoke_service.py b/services/ai_automation_smoke_service.py index 4cf314f..29dc8c9 100644 --- a/services/ai_automation_smoke_service.py +++ b/services/ai_automation_smoke_service.py @@ -212,6 +212,8 @@ def build_scheduled_automation_health_summary( surface_details = surface_readback.get("details") or {} sitewide_readback = _find_check(source_result, "Sitewide UI/UX Agent readback") sitewide_details = sitewide_readback.get("details") or {} + visual_qa_readback = _find_check(source_result, "Sitewide visual QA readback") + visual_qa_details = visual_qa_readback.get("details") or {} smoke_status = source_result.get("status") or ("warning" if latest_history else "warning") freshness_family = _history_freshness_family( latest_history, @@ -300,6 +302,30 @@ def build_scheduled_automation_health_summary( "writes_database": False, }, }, + { + "key": "sitewide_visual_qa", + "label": "Sitewide visual QA", + "status": visual_qa_readback.get("status") or "warning", + "summary": visual_qa_readback.get("summary") or "Sitewide visual QA has no latest check.", + "next_machine_action": ( + "keep_sitewide_visual_qa_monitoring" + if visual_qa_readback.get("status") == "ok" + else "run_sitewide_visual_qa_and_publish_artifact" + ), + "details": { + "policy": visual_qa_details.get("policy"), + "result_count": int(visual_qa_details.get("result_count") or 0), + "route_count": int(visual_qa_details.get("route_count") or 0), + "viewport_count": int(visual_qa_details.get("viewport_count") or 0), + "failed_count": int(visual_qa_details.get("failed_count") or 0), + "overflow_issue_count": int(visual_qa_details.get("overflow_issue_count") or 0), + "visual_offender_count": int(visual_qa_details.get("visual_offender_count") or 0), + "artifact_path": visual_qa_details.get("artifact_path"), + "artifact_generated_at": visual_qa_details.get("artifact_generated_at"), + "stale": bool(visual_qa_details.get("stale")), + "writes_database": False, + }, + }, freshness_family, { "key": "daily_summary_delivery", @@ -938,6 +964,61 @@ def _sitewide_ui_ux_agent_check() -> Dict[str, Any]: ) +def _sitewide_visual_qa_check() -> Dict[str, Any]: + """Read-only sentinel for latest desktop/tablet/mobile visual QA artifact.""" + try: + from services.ai_surface_html_readback_service import build_sitewide_visual_qa_readback + + readback = build_sitewide_visual_qa_readback() + summary = readback.get("summary") or {} + status = readback.get("status") or "warning" + failed_count = int(summary.get("failed_count") or 0) + route_count = int(summary.get("route_count") or 0) + viewport_count = int(summary.get("viewport_count") or 0) + summary_text = ( + f"Sitewide visual QA ok,{route_count} routes x {viewport_count} viewports 無 overflow 退化" + if status == "ok" + else ( + f"Sitewide visual QA 需更新或修復:failed={failed_count}、" + f"routes={route_count}、stale={bool(summary.get('stale'))}" + ) + ) + return _check( + "Sitewide visual QA readback", + status, + summary_text, + { + "policy": readback.get("policy"), + "result_count": int(summary.get("result_count") or 0), + "route_count": route_count, + "viewport_count": viewport_count, + "pass_count": int(summary.get("pass_count") or 0), + "failed_count": failed_count, + "overflow_issue_count": int(summary.get("overflow_issue_count") or 0), + "visual_offender_count": int(summary.get("visual_offender_count") or 0), + "artifact_path": readback.get("artifact_path"), + "artifact_exists": bool(readback.get("artifact_exists")), + "artifact_generated_at": readback.get("artifact_generated_at"), + "age_hours": summary.get("age_hours"), + "stale": bool(summary.get("stale")), + "next_machine_action": readback.get("next_machine_action"), + "writes_database": False, + "writes_database_count": 0, + "primary_human_gate_count": 0, + }, + ) + except Exception as exc: + return _check( + "Sitewide visual QA readback", + "critical", + f"Sitewide visual QA readback 無法執行:{exc}", + { + "writes_database": False, + "primary_human_gate_count": 0, + }, + ) + + def collect_ai_automation_smoke(*, record_history: bool = True, history_limit: int = 20) -> Dict[str, Any]: checks: List[Dict[str, Any]] = [ _event_router_check(), @@ -949,6 +1030,7 @@ def collect_ai_automation_smoke(*, record_history: bool = True, history_limit: i _pchome_controlled_apply_drift_monitor_check(), _ai_surface_html_readback_check(), _sitewide_ui_ux_agent_check(), + _sitewide_visual_qa_check(), ] worst = max(checks, key=lambda item: STATUS_RANK.get(item["status"], 2))["status"] result = { diff --git a/services/ai_surface_html_readback_service.py b/services/ai_surface_html_readback_service.py index e8fbb6f..db1bfaa 100644 --- a/services/ai_surface_html_readback_service.py +++ b/services/ai_surface_html_readback_service.py @@ -2,6 +2,8 @@ from __future__ import annotations +import json +import os from dataclasses import dataclass from datetime import datetime from pathlib import Path @@ -14,7 +16,13 @@ POLICY = "read_only_ai_surface_html_readback_v1" REPAIR_POLICY = "read_only_ai_surface_html_repair_package_v1" SITEWIDE_POLICY = "read_only_sitewide_ui_ux_agent_inventory_v1" SITEWIDE_REPAIR_POLICY = "read_only_sitewide_ui_ux_repair_package_v1" +SITEWIDE_VISUAL_QA_POLICY = "read_only_sitewide_visual_qa_readback_v1" ROOT = Path(__file__).resolve().parents[1] +SITEWIDE_VISUAL_QA_ARTIFACT_PATH = os.getenv( + "MOMO_SITEWIDE_VISUAL_QA_ARTIFACT", + str(ROOT / "data" / "ai_automation" / "sitewide_visual_qa_latest.json"), +) +SITEWIDE_VISUAL_QA_MAX_AGE_HOURS = int(os.getenv("MOMO_SITEWIDE_VISUAL_QA_MAX_AGE_HOURS", "26")) @dataclass(frozen=True) @@ -135,6 +143,143 @@ PROFESSIONAL_GUARDRAIL_MARKERS: tuple[str, ...] = ( ) +def _parse_timestamp(value: Any) -> datetime | None: + if not value: + return None + text_value = str(value).strip() + if not text_value: + return None + try: + return datetime.fromisoformat(text_value.replace("Z", "+00:00")) + except ValueError: + return None + + +def _visual_qa_payload_from_results(raw_payload: Any) -> dict[str, Any]: + if isinstance(raw_payload, list): + return {"results": raw_payload} + if isinstance(raw_payload, dict): + return raw_payload + return {"results": []} + + +def _visual_qa_summary(payload: dict[str, Any]) -> dict[str, Any]: + results = list(payload.get("results") or []) + failed = [item for item in results if not item.get("passed")] + routes = sorted({str(item.get("route") or "") for item in results if item.get("route")}) + viewports = sorted({str(item.get("viewport") or "") for item in results if item.get("viewport")}) + overflow_issues = [ + item for item in failed + if "overflow" in str(item.get("error") or "").lower() + ] + visual_offenders = [ + item for item in failed + if "offender" in str(item.get("error") or "").lower() + ] + return { + "result_count": len(results), + "route_count": len(routes), + "viewport_count": len(viewports), + "pass_count": len(results) - len(failed), + "failed_count": len(failed), + "overflow_issue_count": len(overflow_issues), + "visual_offender_count": len(visual_offenders), + "routes": routes, + "viewports": viewports, + } + + +def build_sitewide_visual_qa_readback( + *, + artifact_path: Path | str | None = None, + artifact_payload: dict[str, Any] | list[dict[str, Any]] | None = None, + max_age_hours: int | None = None, +) -> dict[str, Any]: + """Read the latest Playwright responsive QA artifact without running a browser.""" + path = Path(artifact_path or SITEWIDE_VISUAL_QA_ARTIFACT_PATH) + max_age = max(1, int(max_age_hours or SITEWIDE_VISUAL_QA_MAX_AGE_HOURS)) + read_error = "" + payload: dict[str, Any] + if artifact_payload is not None: + payload = _visual_qa_payload_from_results(artifact_payload) + source_kind = "inline_payload" + artifact_exists = True + else: + source_kind = "artifact_file" + artifact_exists = path.exists() + if not artifact_exists: + payload = {"results": []} + read_error = f"artifact_not_found:{path}" + else: + try: + payload = _visual_qa_payload_from_results(json.loads(path.read_text(encoding="utf-8"))) + except (OSError, json.JSONDecodeError) as exc: + payload = {"results": []} + read_error = str(exc)[:300] + + summary = _visual_qa_summary(payload) + generated_at = payload.get("generated_at") or payload.get("completed_at") + generated_dt = _parse_timestamp(generated_at) + age_hours = None + stale = False + if generated_dt is not None: + now = datetime.now(generated_dt.tzinfo) if generated_dt.tzinfo else datetime.now() + age_hours = max((now - generated_dt).total_seconds() / 3600, 0) + stale = age_hours > max_age + + status = "ok" + if read_error or summary["result_count"] == 0: + status = "warning" + elif summary["failed_count"] > 0: + status = "critical" + elif generated_dt is None or stale: + status = "warning" + + failed_results = [ + item for item in list(payload.get("results") or []) + if not item.get("passed") + ][:12] + return { + "policy": SITEWIDE_VISUAL_QA_POLICY, + "status": status, + "version": SYSTEM_VERSION, + "generated_at": datetime.now().isoformat(timespec="seconds"), + "source_kind": source_kind, + "artifact_path": str(path), + "artifact_exists": artifact_exists, + "artifact_generated_at": generated_at, + "summary": { + **summary, + "age_hours": round(age_hours, 2) if age_hours is not None else None, + "max_age_hours": max_age, + "stale": stale, + "read_error": read_error, + "primary_human_gate_count": 0, + "writes_database_count": 0, + }, + "failed_results": failed_results, + "next_machine_action": ( + "keep_sitewide_visual_qa_monitoring" + if status == "ok" + else "run_sitewide_visual_qa_and_publish_artifact" + ), + "automation_policy": { + "primary_flow": "ai_controlled", + "manual_review_mode": "exception_only", + "machine_verifiable_evidence": True, + "primary_human_gate_count": 0, + }, + "safety": { + "read_only": True, + "writes_database": False, + "writes_database_count": 0, + "sends_notifications": False, + "requires_browser_for_readback": False, + "requires_browser_for_artifact_generation": True, + }, + } + + def _read_surface_html(root: Path, contract: SurfaceContract) -> str: return (root / contract.template).read_text(encoding="utf-8") diff --git a/tests/test_ai_automation_metrics.py b/tests/test_ai_automation_metrics.py index 0e943c7..0971f09 100644 --- a/tests/test_ai_automation_metrics.py +++ b/tests/test_ai_automation_metrics.py @@ -107,10 +107,10 @@ def test_system_metrics_exports_scheduled_health_summary(): Gauge, { "summary": { - "ok": 3, + "ok": 4, "warning": 1, "critical": 0, - "total": 4, + "total": 5, "primary_human_gate_count": 0, "writes_database_count": 0, }, @@ -118,12 +118,13 @@ def test_system_metrics_exports_scheduled_health_summary(): {"key": "ai_automation_smoke", "status": "ok"}, {"key": "pchome_controlled_apply_drift_monitor", "status": "warning"}, {"key": "sitewide_ui_ux_agent", "status": "ok"}, + {"key": "sitewide_visual_qa", "status": "ok"}, ], }, ) output = generate_latest(registry).decode("utf-8") - assert 'momo_ai_automation_scheduled_health_summary_total{status="ok"} 3.0' in output + assert 'momo_ai_automation_scheduled_health_summary_total{status="ok"} 4.0' in output assert 'momo_ai_automation_scheduled_health_summary_total{status="warning"} 1.0' in output assert ( 'momo_ai_automation_scheduled_health_family_status{family="ai_automation_smoke",status="ok"} 1.0' @@ -137,6 +138,10 @@ def test_system_metrics_exports_scheduled_health_summary(): 'momo_ai_automation_scheduled_health_family_status{family="sitewide_ui_ux_agent",status="ok"} 1.0' in output ) + assert ( + 'momo_ai_automation_scheduled_health_family_status{family="sitewide_visual_qa",status="ok"} 1.0' + in output + ) assert "momo_ai_automation_scheduled_health_primary_human_gate_count 0.0" in output assert "momo_ai_automation_scheduled_health_writes_database_count 0.0" in output diff --git a/tests/test_ai_automation_smoke_service.py b/tests/test_ai_automation_smoke_service.py index 422fc86..59c7482 100644 --- a/tests/test_ai_automation_smoke_service.py +++ b/tests/test_ai_automation_smoke_service.py @@ -38,11 +38,12 @@ def test_collect_ai_automation_smoke_uses_worst_status(monkeypatch): monkeypatch.setattr(smoke, "_pchome_controlled_apply_drift_monitor_check", lambda: smoke._check("pchome", "ok", "ok")) monkeypatch.setattr(smoke, "_ai_surface_html_readback_check", lambda: smoke._check("surface", "ok", "ok")) monkeypatch.setattr(smoke, "_sitewide_ui_ux_agent_check", lambda: smoke._check("sitewide", "ok", "ok")) + monkeypatch.setattr(smoke, "_sitewide_visual_qa_check", lambda: smoke._check("visual", "ok", "ok")) result = smoke.collect_ai_automation_smoke(record_history=False) assert result["status"] == "critical" - assert result["summary"] == {"ok": 7, "warning": 1, "critical": 1, "total": 9} + assert result["summary"] == {"ok": 8, "warning": 1, "critical": 1, "total": 10} def test_pchome_controlled_apply_drift_monitor_reports_verified_zero_drift(monkeypatch): @@ -111,6 +112,7 @@ def test_collect_ai_automation_smoke_persists_recent_history(tmp_path, monkeypat monkeypatch.setattr(smoke, "_pchome_controlled_apply_drift_monitor_check", lambda: smoke._check("pchome", "ok", "ok")) monkeypatch.setattr(smoke, "_ai_surface_html_readback_check", lambda: smoke._check("surface", "ok", "ok")) monkeypatch.setattr(smoke, "_sitewide_ui_ux_agent_check", lambda: smoke._check("sitewide", "ok", "ok")) + monkeypatch.setattr(smoke, "_sitewide_visual_qa_check", lambda: smoke._check("visual", "ok", "ok")) first = smoke.collect_ai_automation_smoke(history_limit=5) second = smoke.collect_ai_automation_smoke(history_limit=5) @@ -166,7 +168,7 @@ def test_scheduled_automation_health_summary_reads_history_without_side_effects( json.dumps({ "generated_at": datetime.now().isoformat(timespec="seconds"), "status": "ok", - "summary": {"ok": 9, "warning": 0, "critical": 0, "total": 9}, + "summary": {"ok": 10, "warning": 0, "critical": 0, "total": 10}, "checks": [ { "name": "PChome 受控落地 drift monitor", @@ -206,6 +208,23 @@ def test_scheduled_automation_health_summary_reads_history_without_side_effects( "repair_package_status": "no_op", "repair_action_count": 0, }, + }, + { + "name": "Sitewide visual QA readback", + "status": "ok", + "summary": "Sitewide visual QA ok,12 routes x 3 viewports 無 overflow 退化", + "details": { + "policy": "read_only_sitewide_visual_qa_readback_v1", + "result_count": 36, + "route_count": 12, + "viewport_count": 3, + "failed_count": 0, + "overflow_issue_count": 0, + "visual_offender_count": 0, + "artifact_path": "data/ai_automation/sitewide_visual_qa_latest.json", + "artifact_generated_at": datetime.now().isoformat(timespec="seconds"), + "stale": False, + }, } ], }, ensure_ascii=False) + "\n", @@ -224,7 +243,7 @@ def test_scheduled_automation_health_summary_reads_history_without_side_effects( ) assert summary["policy"] == "read_only_ai_automation_scheduled_health_summary" assert summary["status"] == "ok" - assert summary["summary"]["total"] == 6 + assert summary["summary"]["total"] == 7 assert summary["summary"]["primary_human_gate_count"] == 0 assert summary["summary"]["writes_database_count"] == 0 assert pchome_family["status"] == "ok" @@ -242,6 +261,13 @@ def test_scheduled_automation_health_summary_reads_history_without_side_effects( assert sitewide_family["status"] == "ok" assert sitewide_family["details"]["issue_surface_count"] == 0 assert sitewide_family["details"]["repair_package_status"] == "no_op" + visual_family = next( + item for item in summary["families"] + if item["key"] == "sitewide_visual_qa" + ) + assert visual_family["status"] == "ok" + assert visual_family["details"]["route_count"] == 12 + assert visual_family["details"]["failed_count"] == 0 assert summary["scheduled_outputs"]["telegram_send_in_preview"] is False assert summary["scheduled_outputs"]["writes_database"] is False assert summary["automation_policy"]["primary_flow"] == "ai_controlled" @@ -256,7 +282,7 @@ def test_scheduled_automation_health_summary_can_use_current_smoke_without_recor current_smoke = { "generated_at": "2026-07-02T12:00:00", "status": "critical", - "summary": {"ok": 8, "warning": 0, "critical": 1, "total": 9}, + "summary": {"ok": 9, "warning": 0, "critical": 1, "total": 10}, "checks": [ { "name": "PChome 受控落地 drift monitor", @@ -281,6 +307,19 @@ def test_scheduled_automation_health_summary_can_use_current_smoke_without_recor "repair_package_status": "no_op", "repair_action_count": 0, }, + }, + { + "name": "Sitewide visual QA readback", + "status": "ok", + "summary": "Sitewide visual QA ok,12 routes x 3 viewports 無 overflow 退化", + "details": { + "route_count": 12, + "viewport_count": 3, + "failed_count": 0, + "overflow_issue_count": 0, + "visual_offender_count": 0, + "stale": False, + }, } ], } @@ -307,7 +346,7 @@ def test_scheduled_automation_health_summary_route_returns_compact_payload(tmp_p json.dumps({ "generated_at": datetime.now().isoformat(timespec="seconds"), "status": "ok", - "summary": {"ok": 8, "warning": 0, "critical": 0, "total": 8}, + "summary": {"ok": 10, "warning": 0, "critical": 0, "total": 10}, "checks": [], }, ensure_ascii=False) + "\n", encoding="utf-8", @@ -474,6 +513,58 @@ def test_sitewide_ui_ux_agent_reports_non_utf8_templates(tmp_path): assert package["safety"]["writes_database"] is False +def test_sitewide_visual_qa_readback_consumes_artifact_payload(): + from datetime import datetime + from services.ai_surface_html_readback_service import build_sitewide_visual_qa_readback + + payload = build_sitewide_visual_qa_readback( + artifact_payload={ + "generated_at": datetime.now().isoformat(timespec="seconds"), + "results": [ + {"route": "/ai_intelligence", "viewport": "desktop-1440", "passed": True}, + {"route": "/ai_intelligence", "viewport": "mobile-390", "passed": True}, + {"route": "/observability/overview", "viewport": "mobile-390", "passed": True}, + ], + } + ) + + assert payload["policy"] == "read_only_sitewide_visual_qa_readback_v1" + assert payload["status"] == "ok" + assert payload["summary"]["result_count"] == 3 + assert payload["summary"]["route_count"] == 2 + assert payload["summary"]["viewport_count"] == 2 + assert payload["summary"]["failed_count"] == 0 + assert payload["summary"]["primary_human_gate_count"] == 0 + assert payload["summary"]["writes_database_count"] == 0 + assert payload["safety"]["requires_browser_for_readback"] is False + assert payload["safety"]["requires_browser_for_artifact_generation"] is True + + +def test_sitewide_visual_qa_readback_reports_visual_failures(): + from datetime import datetime + from services.ai_surface_html_readback_service import build_sitewide_visual_qa_readback + + payload = build_sitewide_visual_qa_readback( + artifact_payload={ + "generated_at": datetime.now().isoformat(timespec="seconds"), + "results": [ + { + "route": "/price_comparison", + "viewport": "mobile-390", + "passed": False, + "error": "body horizontal overflow 24px", + } + ], + } + ) + + assert payload["status"] == "critical" + assert payload["summary"]["failed_count"] == 1 + assert payload["summary"]["overflow_issue_count"] == 1 + assert payload["next_machine_action"] == "run_sitewide_visual_qa_and_publish_artifact" + assert payload["failed_results"][0]["route"] == "/price_comparison" + + def test_surface_html_readback_check_is_part_of_ai_smoke(monkeypatch): from services import ai_automation_smoke_service as smoke @@ -484,6 +575,12 @@ def test_surface_html_readback_check_is_part_of_ai_smoke(monkeypatch): monkeypatch.setattr(smoke, "_embedding_queue_check", lambda: smoke._check("embedding", "ok", "ok")) monkeypatch.setattr(smoke, "_elephant_hitl_check", lambda: smoke._check("elephant", "ok", "ok")) monkeypatch.setattr(smoke, "_pchome_controlled_apply_drift_monitor_check", lambda: smoke._check("pchome", "ok", "ok")) + monkeypatch.setattr(smoke, "_sitewide_visual_qa_check", lambda: smoke._check( + "Sitewide visual QA readback", + "ok", + "visual ok", + {"route_count": 12, "viewport_count": 3, "failed_count": 0}, + )) result = smoke.collect_ai_automation_smoke(record_history=False) @@ -495,12 +592,18 @@ def test_surface_html_readback_check_is_part_of_ai_smoke(monkeypatch): item for item in result["checks"] if item["name"] == "Sitewide UI/UX Agent readback" ) - assert result["summary"]["total"] == 9 + visual_check = next( + item for item in result["checks"] + if item["name"] == "Sitewide visual QA readback" + ) + assert result["summary"]["total"] == 10 assert surface_check["status"] == "ok" assert surface_check["details"]["checked_surface_count"] == 10 assert sitewide_check["status"] == "ok" assert sitewide_check["details"]["issue_surface_count"] == 0 assert sitewide_check["details"]["repair_package_status"] == "no_op" + assert visual_check["status"] == "ok" + assert visual_check["details"]["route_count"] == 12 def test_surface_html_readback_route_returns_payload(monkeypatch): @@ -538,6 +641,11 @@ def test_surface_and_sitewide_ui_ux_agent_routes_return_payloads(): routes.ai_automation_sitewide_ui_ux_repair_package_api.__wrapped__, "read_only_sitewide_ui_ux_repair_package_v1", ), + ( + "/api/ai-automation/sitewide-visual-qa-readback?max_age_hours=9999", + routes.ai_automation_sitewide_visual_qa_readback_api.__wrapped__, + "read_only_sitewide_visual_qa_readback_v1", + ), ] for path, view_func, policy in route_cases: