337 lines
10 KiB
Markdown
337 lines
10 KiB
Markdown
# RunBook: E2E Playwright CI 定期排程設定
|
||
|
||
> **類型**: 操作型 RunBook
|
||
> **優先級**: 🔴 P0
|
||
> **建立**: 2026-03-29 12:38 (台北)
|
||
> **建立者**: Antigravity
|
||
> **工時預估**: 30 分鐘
|
||
> **前置條件**: Playwright 測試可在本機 `pnpm test:e2e` 成功執行
|
||
|
||
---
|
||
|
||
## 背景與現況
|
||
|
||
### 🔍 精確現況診斷
|
||
|
||
**已有的 12 個 E2E 測試檔案**（`apps/web/tests/e2e/`）：
|
||
|
||
| 測試檔案 | 測試範圍 |
|
||
|---------|---------|
|
||
| `dashboard-acceptance.spec.ts` | 首頁 Dashboard 驗收 |
|
||
| `multisig-security.spec.ts` | 多重簽核安全性 |
|
||
| `approval-card-verify.spec.ts` | 簽核卡片驗證 |
|
||
| `phase11-conversational.spec.ts` | 對話式 Phase 11 功能 |
|
||
| `phase19-production-verification.spec.ts` | Phase 19 生產驗證 |
|
||
| `action-log.spec.ts` | 行動日誌 |
|
||
| `cpo102-visual.spec.ts` | 視覺截圖測試 |
|
||
| `visual-armor-upgrade.spec.ts` | 視覺升級驗證 |
|
||
| `debug-error.spec.ts` | 錯誤頁面 |
|
||
| `rbac-screenshot.spec.ts` | RBAC 截圖驗證 |
|
||
| `phase4-final-demo.spec.ts` | Phase 4 Demo |
|
||
| `phase4-timeline.spec.ts` | Phase 4 時間軸 |
|
||
|
||
**缺口**：`playwright.config.ts` 已有配置，但 `.github/workflows/` 中**無定期執行排程**。
|
||
|
||
---
|
||
|
||
## Step 1: 建立 E2E 定期排程 Workflow
|
||
|
||
建立 `.github/workflows/e2e-weekly.yaml`：
|
||
|
||
```yaml
|
||
name: 🎭 E2E Playwright 週期驗收
|
||
|
||
on:
|
||
# 每週一凌晨 02:30 執行（台北時間，即 UTC 18:30 週日）
|
||
schedule:
|
||
- cron: '30 18 * * 0'
|
||
|
||
# 允許手動觸發
|
||
workflow_dispatch:
|
||
inputs:
|
||
environment:
|
||
description: '測試環境 URL'
|
||
required: false
|
||
default: 'https://192.168.0.120:32335'
|
||
test_suite:
|
||
description: '測試套件 (all / smoke / visual)'
|
||
required: false
|
||
default: 'all'
|
||
|
||
jobs:
|
||
e2e-test:
|
||
name: 🎭 E2E Playwright (${{ matrix.browser }})
|
||
runs-on: [self-hosted, harbor] # 使用 .110 的 GitHub Runner
|
||
|
||
strategy:
|
||
fail-fast: false # 一個瀏覽器失敗不影響其他
|
||
matrix:
|
||
browser: [chromium, firefox]
|
||
|
||
steps:
|
||
- name: Checkout
|
||
uses: actions/checkout@v4
|
||
|
||
- name: Install pnpm
|
||
uses: pnpm/action-setup@v4
|
||
with:
|
||
version: 9
|
||
|
||
- name: Setup Node.js
|
||
uses: actions/setup-node@v4
|
||
with:
|
||
node-version: '20'
|
||
cache: 'pnpm'
|
||
|
||
- name: Install dependencies
|
||
run: pnpm install --frozen-lockfile
|
||
|
||
- name: Install Playwright Browsers
|
||
run: pnpm --filter web exec playwright install --with-deps ${{ matrix.browser }}
|
||
|
||
- name: 🎭 Run E2E Tests
|
||
run: |
|
||
cd apps/web
|
||
pnpm exec playwright test \
|
||
--project=${{ matrix.browser }} \
|
||
--reporter=html \
|
||
${SUITE_FILTER}
|
||
env:
|
||
SUITE_FILTER: ${{ github.event.inputs.test_suite == 'smoke' && '--grep @smoke' || '' }}
|
||
BASE_URL: ${{ github.event.inputs.environment || 'http://192.168.0.120:32335' }}
|
||
CI: true
|
||
|
||
- name: 📁 Upload Test Report
|
||
uses: actions/upload-artifact@v4
|
||
if: always()
|
||
with:
|
||
name: playwright-report-${{ matrix.browser }}-${{ github.run_id }}
|
||
path: apps/web/playwright-report/
|
||
retention-days: 14
|
||
|
||
- name: 📸 Upload Screenshots on Failure
|
||
uses: actions/upload-artifact@v4
|
||
if: failure()
|
||
with:
|
||
name: e2e-screenshots-${{ matrix.browser }}-${{ github.run_id }}
|
||
path: apps/web/test-results/
|
||
retention-days: 7
|
||
|
||
- name: 🚨 Notify Telegram on Failure
|
||
if: failure()
|
||
run: |
|
||
curl -s -X POST "https://api.telegram.org/bot${TG_BOT_TOKEN}/sendMessage" \
|
||
-d chat_id="${TG_CHAT_ID}" \
|
||
-d parse_mode="HTML" \
|
||
-d text="🎭 <b>E2E 週期測試失敗</b>
|
||
|
||
瀏覽器：${{ matrix.browser }}
|
||
觸發：${{ github.event_name }}
|
||
時間：$(TZ='Asia/Taipei' date '+%Y-%m-%d %H:%M:%S')
|
||
|
||
報告：${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
|
||
|
||
🔍 請立即調查 UI 回歸問題"
|
||
env:
|
||
TG_BOT_TOKEN: ${{ secrets.OPENCLAW_TG_BOT_TOKEN }}
|
||
TG_CHAT_ID: ${{ env.TELEGRAM_ALERT_CHAT_ID }}
|
||
|
||
# 視覺截圖比對工作（獨立 job，只在排程時執行）
|
||
visual-regression:
|
||
name: 📸 視覺回歸比對
|
||
runs-on: [self-hosted, harbor]
|
||
if: github.event_name == 'schedule' # 只在定期排程時執行
|
||
needs: e2e-test
|
||
|
||
steps:
|
||
- name: Checkout
|
||
uses: actions/checkout@v4
|
||
|
||
- name: Install pnpm & deps
|
||
run: |
|
||
pnpm install --frozen-lockfile
|
||
pnpm --filter web exec playwright install --with-deps chromium
|
||
|
||
- name: 📸 Generate Visual Snapshots
|
||
run: |
|
||
cd apps/web
|
||
pnpm exec playwright test \
|
||
--project=chromium \
|
||
--grep @visual \
|
||
--reporter=html \
|
||
--update-snapshots=missing # 新截圖自動建立 baseline
|
||
env:
|
||
BASE_URL: 'http://192.168.0.120:32335'
|
||
CI: true
|
||
|
||
- name: 📁 Upload Visual Snapshots
|
||
uses: actions/upload-artifact@v4
|
||
with:
|
||
name: visual-snapshots-${{ github.run_id }}
|
||
path: apps/web/tests/e2e/__snapshots__/
|
||
retention-days: 30
|
||
|
||
- name: 🚨 Notify Telegram on Visual Regression
|
||
if: failure()
|
||
run: |
|
||
curl -s -X POST "https://api.telegram.org/bot${TG_BOT_TOKEN}/sendMessage" \
|
||
-d chat_id="${TG_CHAT_ID}" \
|
||
-d parse_mode="HTML" \
|
||
-d text="📸 <b>視覺回歸測試失敗</b>
|
||
|
||
AWOOOI 前端出現視覺變化！
|
||
時間：$(TZ='Asia/Taipei' date '+%Y-%m-%d %H:%M:%S')
|
||
|
||
請統帥審查截圖比對報告：
|
||
${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
|
||
env:
|
||
TG_BOT_TOKEN: ${{ secrets.OPENCLAW_TG_BOT_TOKEN }}
|
||
TG_CHAT_ID: ${{ env.TELEGRAM_ALERT_CHAT_ID }}
|
||
```
|
||
|
||
---
|
||
|
||
## Step 2: 標記現有測試為 Smoke / Visual 類別
|
||
|
||
```typescript
|
||
// 在各 spec 檔案第一行測試加上標籤：
|
||
|
||
// dashboard-acceptance.spec.ts（核心功能，標記為 @smoke）
|
||
test.describe('Dashboard Acceptance @smoke', () => { ... });
|
||
|
||
// cpo102-visual.spec.ts（視覺截圖，標記為 @visual）
|
||
test.describe('Visual Regression @visual', () => {
|
||
test('CPO-102 首頁視覺', async ({ page }) => {
|
||
await page.goto('/');
|
||
await expect(page).toHaveScreenshot('dashboard-baseline.png', {
|
||
fullPage: true,
|
||
threshold: 0.02 // 允許 2% 像素差異
|
||
});
|
||
});
|
||
});
|
||
```
|
||
|
||
**建議標記方式**：
|
||
|
||
| 標籤 | 包含測試 | 執行頻率 |
|
||
|------|---------|---------|
|
||
| `@smoke` | dashboard, approval, action-log | 每次 CD 後 + 每週 |
|
||
| `@visual` | cpo102-visual, visual-armor | 只在每週排程 |
|
||
| （無標籤） | 所有其他測試 | 只在每週排程 |
|
||
|
||
---
|
||
|
||
## Step 3: 部署並驗證
|
||
|
||
```bash
|
||
# 1. 提交 Workflow 檔案
|
||
git add .github/workflows/e2e-weekly.yaml
|
||
git commit -m "feat(ci): add weekly E2E Playwright schedule with Telegram failure notification"
|
||
git push origin main
|
||
|
||
# 2. 手動觸發測試（確認 Workflow 運作正常）
|
||
gh workflow run e2e-weekly.yaml \
|
||
-f environment="http://192.168.0.120:32335" \
|
||
-f test_suite="smoke"
|
||
|
||
# 3. 監控 Workflow 執行
|
||
gh run watch
|
||
|
||
# 4. 確認 Telegram Bot 收到失敗通知（刻意讓一個測試失敗）
|
||
```
|
||
|
||
---
|
||
|
||
## 驗收標準
|
||
|
||
| 項目 | 通過條件 |
|
||
|------|---------|
|
||
| Workflow 存在 | `.github/workflows/e2e-weekly.yaml` 成功 push |
|
||
| 手動觸發正常 | `gh workflow run` 可執行且完成 |
|
||
| Smoke 測試通過 | `@smoke` 標籤測試全部 PASS |
|
||
| 失敗通知正常 | Telegram Bot 收到失敗訊息 |
|
||
| 報告上傳 | GitHub Actions Artifacts 中有 `playwright-report-*` |
|
||
|
||
---
|
||
|
||
## ⚠️ 架構安全補丁（2026-03-29 更新，部署前必讀）
|
||
|
||
> 來源：`ARCHITECTURAL_RISK_WAR_GAME.md` 深度沙盤推演，代碼確認級別
|
||
|
||
### 補丁 1：playwright.config.ts 必須加入 ignoreHTTPSErrors
|
||
|
||
**問題**：內網 K3s 使用自簽憑證（Self-signed cert），Playwright 連接 `https://192.168.0.120:32335` 時會遭遇 `NET::ERR_CERT_AUTHORITY_INVALID`，導致**所有測試在第一步就失敗**。
|
||
|
||
**修復**（`apps/web/playwright.config.ts`）：
|
||
|
||
```typescript
|
||
export default defineConfig({
|
||
use: {
|
||
baseURL: process.env.BASE_URL || 'http://192.168.0.120:32335',
|
||
ignoreHTTPSErrors: true, // 🆕 必須加入，否則自簽憑證全面阻擋
|
||
viewport: { width: 1280, height: 720 },
|
||
deviceScaleFactor: 1, // 🆕 防止 Retina 螢幕 DPI 差異影響截圖比對
|
||
},
|
||
expect: {
|
||
toHaveScreenshot: {
|
||
threshold: 0.05, // 🆕 允許 5% 差異（吸收跨平台字體渲染微差）
|
||
maxDiffPixelRatio: 0.05,
|
||
},
|
||
},
|
||
});
|
||
```
|
||
|
||
---
|
||
|
||
### 補丁 2：Visual Baseline 必須在 Docker（Linux 環境）中產生
|
||
|
||
**問題**：Mac（CoreText 渲染）與 GitHub Actions CI（Linux FreeType 渲染）的字體像素不同。若在 Mac 本機產生 baseline，CI 比對時**100% 誤報失敗**。
|
||
|
||
> 🔴 **絕對禁止在本機 Mac 環境執行 `--update-snapshots`！**
|
||
|
||
**正確的 Baseline 更新流程**：
|
||
|
||
```bash
|
||
# 在 Mac 本機執行，但透過 Docker（Linux 環境）產生截圖
|
||
cd apps/web
|
||
docker run --rm \
|
||
-v $(pwd):/work -w /work \
|
||
-p 3000:3000 \
|
||
mcr.microsoft.com/playwright:v1.44.0-jammy \
|
||
pnpm exec playwright test \
|
||
--update-snapshots \
|
||
--project=chromium \
|
||
--grep @visual
|
||
|
||
# Docker 產生的 .png 自動存入 tests/e2e/__snapshots__/
|
||
# 提 PR，標注 📸 VISUAL_UPDATE
|
||
# 統帥視覺審核截圖後方可合併
|
||
```
|
||
|
||
**加入 package.json scripts**：
|
||
|
||
```json
|
||
{
|
||
"scripts": {
|
||
"test:visual": "playwright test --project=chromium --grep @visual",
|
||
"test:visual:update": "docker run --rm -v $(pwd):/work -w /work mcr.microsoft.com/playwright:v1.44.0-jammy pnpm exec playwright test --update-snapshots --project=chromium --grep @visual"
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
### 補丁 3：CI Threshold 需與本 RunBook Step 2 標籤保持一致
|
||
|
||
Visual 測試中 `threshold: 0.02`（Step 2 示例代碼）與 `playwright.config.ts` 全局設定 `0.05` 會以**個別設定優先**。建議統一為：
|
||
|
||
```typescript
|
||
// 全局（playwright.config.ts）
|
||
threshold: 0.05 // 寬鬆（跨平台環境差異）
|
||
|
||
// 個別敏感組件（*.spec.ts）
|
||
await expect(page).toHaveScreenshot('component.png', {
|
||
threshold: 0.02 // 嚴格（關鍵組件精確比對）
|
||
});
|
||
```
|