feat(crawler): expand pixelrag marketplace automation
This commit is contained in:
@@ -54,7 +54,7 @@ python scripts/tools/browse_sh_probe.py -- screenshot
|
||||
|
||||
## PixelRAG-style 視覺證據 fallback
|
||||
|
||||
2026-07-09 評估 PixelRAG 後,結論是「可導入,但不可直接取代正式爬蟲」。PixelRAG 的核心價值是把渲染後頁面截圖切成 tiles,讓 AI 讀到 HTML parser 可能丟失的視覺結構;本專案第一階段只採用這個視覺證據思路,不直接拉外部 runtime、不用外部 embedding API、不把像素結果寫入正式價格表。
|
||||
2026-07-09 評估 PixelRAG 後,結論是「可導入,而且不應只限 MOMO/PChome」。PixelRAG 的核心價值是把渲染後頁面截圖切成 tiles,讓 AI 讀到 HTML parser 可能丟失的視覺結構;本專案第一階段採用這個視覺證據思路,擴充成多電商 public marketplace visual evidence layer,不直接拉外部 runtime、不用外部 embedding API、不把像素結果寫入正式價格表。
|
||||
|
||||
導入順序:
|
||||
|
||||
@@ -65,11 +65,44 @@ python scripts/tools/browse_sh_probe.py -- screenshot
|
||||
5. Phase 4 若需要索引,優先設計 pgvector-compatible evidence metadata;FAISS 只能先當本地研究/ADR 候選,不直接進 production。
|
||||
6. Phase 5 才談 crawler fusion;正式 `competitor_prices` / `competitor_price_history` 寫入仍需 matcher replay/canary 證據。
|
||||
|
||||
多電商優先順序:
|
||||
|
||||
| Priority | Platform | PixelRAG 用途 | 邊界 |
|
||||
|---|---|---|---|
|
||||
| P0 | MOMO / PChome | 既有 structured crawler 失敗時自動產生 visual manifest queue | 正式價格仍以既有 parser/API/matcher 為準 |
|
||||
| P1 | Shopee Taiwan / Coupang Taiwan | 抓公共搜尋頁商品卡、促銷 badge、店家/配送/評分視覺證據 | 不登入、不碰購物車、不繞過 anti-bot |
|
||||
| P2 | Yahoo Shopping / ETMall | 補促銷券、快速到貨、活動價格等視覺訊號 | 先做 controlled probe,再決定 parser |
|
||||
| P3 | friDay / Rakuten Taiwan | 先建立 probe profile 與 search manifest | URL pattern 需 runtime receipt 驗證後再排程 |
|
||||
|
||||
已支援的 platform code:
|
||||
|
||||
```text
|
||||
momo
|
||||
pchome
|
||||
shopee_tw
|
||||
coupang_tw
|
||||
yahoo_shopping_tw
|
||||
etmall_tw
|
||||
friday_tw
|
||||
rakuten_tw
|
||||
```
|
||||
|
||||
自動觸發:
|
||||
|
||||
- `PChomeCrawler.fetch_region_page` 找不到商品 ID 時會 queue PixelRAG manifest。
|
||||
- `PChomeCrawler.fetch_product_details` API 回傳但無可解析商品時會 queue 商品頁 manifest。
|
||||
- `PChomeCrawler.search_products` 搜尋無商品 ID 或 request error 時會 queue 公開搜尋頁 manifest。
|
||||
- queue 預設路徑:正式 app 容器 `/app/data/ai_automation/pixelrag_manifest_queue`,本機為 `runtime_artifacts/pixelrag_manifest_queue`。
|
||||
- queue 只寫 artifact,不寫 DB、不寫正式價格、不讀 secret;capture worker 後續消費 manifest 產生 screenshot/tile receipt。
|
||||
|
||||
機器可讀評估:
|
||||
|
||||
```bash
|
||||
python scripts/ops/report_pixelrag_crawler_integration.py
|
||||
python scripts/ops/report_pixelrag_crawler_integration.py --platform momo
|
||||
python scripts/ops/report_pixelrag_crawler_integration.py --ecommerce-expansion-plan
|
||||
python scripts/ops/report_pixelrag_crawler_integration.py --platform shopee_tw --marketplace-keyword "防曬乳"
|
||||
python scripts/ops/report_pixelrag_crawler_integration.py --platform coupang_tw --marketplace-keyword "iphone"
|
||||
python scripts/ops/report_pixelrag_crawler_integration.py --capability ollama_multimodal_ready --capability pgvector_visual_ready
|
||||
python scripts/ops/report_pixelrag_crawler_integration.py \
|
||||
--platform momo \
|
||||
@@ -84,6 +117,9 @@ python scripts/ops/report_pixelrag_crawler_integration.py \
|
||||
|
||||
```text
|
||||
/api/ai-automation/pixelrag-crawler-integration?platform=momo
|
||||
/api/ai-automation/pixelrag-crawler-integration?expansion_plan=true
|
||||
/api/ai-automation/pixelrag-crawler-integration?platform=shopee_tw&marketplace_keyword=%E9%98%B2%E6%9B%AC%E4%B9%B3
|
||||
/api/ai-automation/pixelrag-crawler-integration?platform=coupang_tw&marketplace_keyword=iphone
|
||||
/api/ai-automation/pixelrag-crawler-integration?platform=pchome&manifest_url=https://24h.pchome.com.tw/prod/TEST-000000001&crawler=PChomeCrawler.search_products&trigger_reason=parser_empty
|
||||
/api/ai-automation/pixelrag-visual-evidence-readback?platform=pchome&manifest_id=4a93e95e5afb414bc8c3
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user