375 lines
24 KiB
Markdown
375 lines
24 KiB
Markdown
# 主機重啟後一頁式總檢查
|
||
|
||
> Version: v1.18
|
||
> Last updated: 2026-06-27 Asia/Taipei
|
||
> Scope: 110 / 120 / 121 / 188 post-reboot service recovery. 112 Kali / Wazuh / active scan 不屬於本流程。
|
||
|
||
---
|
||
|
||
## 1. 使用時機
|
||
|
||
每次 110 / 120 / 121 / 188 任一台主機開機、關機、重啟、斷電恢復、VMware console fsck、Docker / K3s 大量重排後,都先跑本頁,再決定是否宣稱恢復。
|
||
|
||
最新基準:2026-06-27 02:42 live revalidation / backup core recovery。`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` artifact `/tmp/awoooi-post-reboot-readiness-20260627-024151/summary.txt` 回傳 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_PASS=38`、`POST_START_WARN=3`、`POST_START_BLOCKED=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`STOCK_FRESHNESS_STATUS=ok`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`STOCK_BLOCKERS=none`、`BACKUP_CORE_GREEN=1`、`DR_ESCROW_BLOCKED=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_HYGIENE_BLOCKED=0`、`HOST_188_RESULT=HOST_188_HYGIENE_GREEN.`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`WAZUH_COVERAGE_SCOPE=6`、`WAZUH_DIRECT_ACTIVE=2`、`WAZUH_NO_TRANSPORT=1`、`WAZUH_SSH_BLOCKED=3`、`WAZUH_ROUTE_CODE=200`、`WAZUH_TRANSPORT_COUNT=6`、`WAZUH_DASHBOARD_API_CONNECTION=pending_or_spinning`、`WAZUH_DASHBOARD_INDEX_OK=3`、`RUNTIME_ACTION_AUTHORIZED=0`、`OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。本輪實際修復 188 `momo_pg_daily` backup configured drift:先前 00:16 summary 因 `configured_missing_188=1` 暫時 blocked;00:19 已備份 188 crontab 到 `/home/ollama/momo_backups/crontab-before-momo-pg-host-owned-20260627-001925.txt`,並把 MOMO PostgreSQL daily backup 收斂到 host-owned `/home/ollama/bin/momo-pg-backup.sh`;刷新 exporter 後 `configured_missing_188=0`,00:56 `backup-status` 回 `core_blockers=0`。02:41 DR checklist 回 `CORE_COLD_START_GREEN=1`、`RECOVERY_STATE=CORE_READY_DR_OFFSITE_PENDING`;Prometheus recovery contract 回 `awoooi_recovery_core_ready=1`、`awoooi_recovery_dr_offsite_ready=0`。同一輪後續 `post-reboot-declaration-guard.py`、`post-reboot-next-gate-dispatch.sh`、`post-reboot-next-gate-owner-packets.py`、`post-reboot-owner-packet-contract-guard.py`、`post-reboot-owner-response-preflight.py` 必須使用這份 `summary.txt` 或由它產生的 dispatch / packet,不得混用多次 live probe 的不同時間點結果。`NEXT_REQUIRED_GATES=credential_escrow_evidence,wazuh_manager_registry_export` 仍是唯一目前 next gates;DR 仍因 `escrow_missing=5` 不可宣稱 complete;Wazuh manager registry accepted 仍是 `0`,不可把 route `200`、transport `6`、Dashboard index pattern `3` 或脫敏 registry 計數當成全主機納管完成。v1.18 維持 route/API warmup classifier:delegated cold-start 若只因 public route 單次 502 / TLS readback,或 K3s rollout 瞬間單次 `BLOCKED AWOOOI API not reachable`,但 wrapper route retry 已確認 public AWOOOI API health 為 2xx,該 blocker 會降級為 evidence warning;public API 仍失敗、其他 non-route blocker 或 retry 後未恢復仍為 hard blocked。
|
||
|
||
本頁只回答四件事:
|
||
|
||
1. 主機是否開起來。
|
||
2. 服務是否真的可用。
|
||
3. 資料與備份是否新鮮。
|
||
4. 有哪些不能宣稱完成。
|
||
|
||
---
|
||
|
||
## 2. 絕對判定規則
|
||
|
||
| 層級 | 可以宣稱 | 必要證據 |
|
||
|------|----------|----------|
|
||
| `HOST_BOOTED` | 主機已開機 | ping / SSH port 回應,或 console login prompt。 |
|
||
| `HOST_READY` | 主機可管理 | SSH read-only 可登入,failed units / disk / clock / network 無硬阻塞。 |
|
||
| `SERVICE_READY` | 單站服務可用 | route / container / local health / DB 或依賴健康都通過。 |
|
||
| `FULL_STACK_GREEN` | 本輪重啟服務恢復完成 | cold-start `WARN=0` 且 `BLOCKED=0`,route、K3s、DB freshness、backup、alert、CronJob、exporter 都通過。 |
|
||
| `DR_COMPLETE` | 災難復原也完成 | `FULL_STACK_GREEN` 加上 credential escrow missing `0`、offsite / restore / escrow evidence 完整。 |
|
||
|
||
禁止用單一訊號取代整體判定:
|
||
|
||
- 網站 `200` 不等於資料最新。
|
||
- container `healthy` 不等於 DB / backup / alert 正常。
|
||
- K3s node `Ready` 不等於 workload 分散與 CronJob freshness 正常。
|
||
- Wazuh route `200` 不等於所有主機 agent registry accepted。
|
||
- backup fresh 不等於 DR complete;credential escrow 缺口必須獨立保留。
|
||
|
||
---
|
||
|
||
## 3. 10 分鐘只讀總檢查順序
|
||
|
||
優先使用 machine-readable summary:
|
||
|
||
```bash
|
||
scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color
|
||
```
|
||
|
||
此 summary 只做 read-only 檢查,會委派一頁式總檢查、188 host hygiene checklist 與 Wazuh repo-side no-false-green gates,並將 delegated logs 與可重放的 `summary.txt` 保留在 `/tmp/awoooi-post-reboot-readiness-*`。第一眼先看這些欄位:
|
||
|
||
- `SERVICE_GREEN=1`:服務面可宣稱恢復。
|
||
- `PRODUCT_DATA_GREEN=1`:MOMO / StockPlatform 主要資料 freshness 可宣稱恢復。
|
||
- `BACKUP_CORE_GREEN=1`:備份核心可宣稱恢復。
|
||
- `DR_ESCROW_BLOCKED=1` / `ESCROW_MISSING_COUNT>0`:不可宣稱 DR complete。
|
||
- `HOST_188_HYGIENE_BLOCKED=0`:188 host hygiene 已收斂;若未來回到 `1`,再走 188 專用 runbook。
|
||
- `WAZUH_MANAGER_REGISTRY_ACCEPTED=0`:不可宣稱 Wazuh 全主機納管恢復。
|
||
- `WAZUH_ROUTE_CODE=200` / `WAZUH_TRANSPORT_COUNT>0` 只能代表 route / transport evidence;仍必須搭配 `WAZUH_COVERAGE_SCOPE`、`WAZUH_DIRECT_ACTIVE`、`WAZUH_NO_TRANSPORT`、`WAZUH_SSH_BLOCKED`、`WAZUH_DASHBOARD_API_CONNECTION` 與 `WAZUH_MANAGER_REGISTRY_ACCEPTED` 判讀。
|
||
- `RUNTIME_ACTION_AUTHORIZED=0`:本流程沒有授權 runtime 寫操作。
|
||
- `OVERALL_DECLARATION`:本輪可使用的最高宣告。
|
||
|
||
summary 顯示 `SERVICE_GREEN=1` 後,先跑宣告 guard,確認本輪可以講什麼、不能講什麼:
|
||
|
||
```bash
|
||
scripts/reboot-recovery/post-reboot-declaration-guard.py --no-color
|
||
```
|
||
|
||
若已經有 `$ARTIFACT_DIR/summary.txt`,同一輪建議固定使用它,避免重跑 live probes 時被 CI / route warmup 瞬間狀態影響:
|
||
|
||
```bash
|
||
SUMMARY_FILE=/tmp/awoooi-post-reboot-readiness-YYYYMMDD-HHMMSS/summary.txt
|
||
scripts/reboot-recovery/post-reboot-declaration-guard.py --no-color --summary-file "$SUMMARY_FILE"
|
||
```
|
||
|
||
這支 guard 只讀取 summary evidence,並輸出本輪允許與禁止宣稱的邊界。若要測試某個說法是否允許,可用 `--proposed`:
|
||
|
||
```bash
|
||
scripts/reboot-recovery/post-reboot-declaration-guard.py --no-color --proposed DR_COMPLETE --proposed WAZUH_REGISTRY_RECOVERED
|
||
```
|
||
|
||
在目前基準下,`DR_COMPLETE`、`WAZUH_REGISTRY_RECOVERED`、`RUNTIME_ACTION_AUTHORIZED` 都必須被拒絕;任何人、任何 AI agent 或任何文件若提出這些宣告,必須先補對應 evidence / owner acceptance。`HOST_188_FULLY_GREEN` 只有在 summary 維持 `HOST_188_HYGIENE_BLOCKED=0` 且 checklist `HOST_188_HYGIENE_GREEN` 時才可宣稱。
|
||
|
||
summary 顯示 `SERVICE_GREEN=1` 但仍有 `NEXT_REQUIRED_GATES` 時,再跑下一 Gate 派工 checklist:
|
||
|
||
```bash
|
||
scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color
|
||
```
|
||
|
||
同一輪 evidence chain 應改用同一份 summary:
|
||
|
||
```bash
|
||
scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color --summary-file "$SUMMARY_FILE" | tee /tmp/awoooi-post-reboot-dispatch.txt
|
||
```
|
||
|
||
這支腳本只把目前 live summary 內的 `NEXT_REQUIRED_GATES` 轉成 owner / required evidence / forbidden action / done criteria。2026-06-26 12:13 起通常只剩 `credential_escrow_evidence` 與 `wazuh_manager_registry_export`;若未來 188 又紅,才會重新出現 `host_188_hygiene_maintenance_window`。它不送 request、不讀 secret、不寫 marker、不 restart / reload / repair / import / delete / patch,也不授權 host / Wazuh / Nginx / K8s / DB runtime action。若它輸出 `SERVICE_GREEN=0`,先回到服務恢復,不進入 boundary dispatch。
|
||
|
||
若要交給 AI / 工單 / owner review 使用,產生機器可讀 owner packet:
|
||
|
||
```bash
|
||
scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color --summary-file "$ARTIFACT_DIR/summary.txt"
|
||
```
|
||
|
||
輸出 JSON 只能作為 intake / review packet,不是 request sent。必須看到 `request_sent_count=0`、`owner_response_accepted_count=0`、`runtime_action_authorized_count=0`,否則視為不合格。
|
||
|
||
送入任何 owner review queue 前,必須先把 JSON 存成 artifact 並跑 contract guard:
|
||
|
||
```bash
|
||
scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color --summary-file "$ARTIFACT_DIR/summary.txt" --output /tmp/awoooi-post-reboot-owner-packets.json
|
||
scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json
|
||
```
|
||
|
||
guard 必須輸出 `POST_REBOOT_OWNER_PACKET_CONTRACT_GUARD_OK gates=<live_next_gate_count> request_sent=0 accepted=0 runtime_gate=0`。目前預期 `gates=2`;若 188 hygiene 回到 blocked,才會是 `gates=3`。若 gate 數量、P0 gate id、`0 / false` 欄位、禁用 secret payload、Wazuh 禁用 active response / host write,或 no-false-green 規則任何一項漂移,視為 `BLOCKED`,不得送 owner request、不得寫 escrow marker、不得進維護窗口、不得宣稱 DR / Wazuh 完成。
|
||
|
||
收到 owner response 檔案前,或收到任何聲稱已補證據的 JSON 前,必須跑 owner response preflight:
|
||
|
||
```bash
|
||
scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --summary-file "$ARTIFACT_DIR/summary.txt"
|
||
scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --owner-packet-file /tmp/awoooi-post-reboot-owner-packets.json
|
||
scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --owner-packet-file /tmp/awoooi-post-reboot-owner-packets.json --response-file docs/templates/post-reboot-next-gate-owner-response.json
|
||
```
|
||
|
||
第一個命令必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_file expected_gates=2 received=0 accepted=0 runtime_gate=0`。第二個命令必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_content expected_gates=2 received=0 accepted=0 runtime_gate=0`,證明空模板不能被算成已收件或已接受。合格 response 只能包含脫敏 evidence refs、owner role / team / decision / reviewer / followup owner、五個 escrow item 的 non-secret evidence ref,以及 Wazuh manager registry / Dashboard API readback;不得包含密碼、token、secret value、hash、prefix/suffix、raw Wazuh payload、agent 原名、內網 IP、`client.keys`、active response、host write、agent re-enroll、Wazuh restart、Kali active scan 或 credential marker write。preflight 通過也只代表可進入獨立 reviewer acceptance,不代表 `DR_COMPLETE`、`WAZUH_REGISTRY_RECOVERED` 或任何 runtime action 授權。
|
||
|
||
需要展開細節時,再使用 repo-side wrapper:
|
||
|
||
```bash
|
||
scripts/reboot-recovery/post-start-quick-check.sh --no-color
|
||
```
|
||
|
||
此 wrapper 只做 read-only 檢查,並委派既有 cold-start / MOMO preflight / backup-status;不 restart、不 reload、不 import、不改 K8s、不讀 token 內容。wrapper 會把 warning 分成 `SERVICE`、`BOUNDARY`、`EVIDENCE` 三類,避免把 `escrow_missing>0` 誤判成服務降級。若 summary 或 wrapper 因某個 SSH 權限或路徑失敗,再依下列分段命令手動補證據。
|
||
|
||
Public route gate 自 v1.6 起會使用 `ROUTE_RETRY_ATTEMPTS`(預設 `3`)與 `ROUTE_RETRY_DELAY_SECONDS`(預設 `2`)重試。StockPlatform dedicated freshness gate 自 v1.13 起使用 `STOCK_FRESHNESS_RETRY_ATTEMPTS`(預設 `6`)與 `STOCK_FRESHNESS_RETRY_DELAY_SECONDS`(預設 `5`),因為它常在 110 Docker / CI rollout 後比 route health 多需要數十秒 warmup。單次 `000` / timeout / `502` 若 retry 後恢復,應列為 evidence warning 或 transient evidence,不可直接當成網站或資料仍壞;只有連續失敗才是 service blocker。
|
||
|
||
Credential escrow gate 自 v1.6 起在 `escrow_missing>0` 時,會只讀呼叫 `/backup/scripts/mark-credential-escrow-verified.sh --status` 並列出缺項。這只是 evidence readback,不會寫 marker、不會讀密碼、不會降低 DR blocker;用途是讓 operator 立即知道缺的是哪幾個非 secret evidence marker。
|
||
|
||
Wrapper 必須先解析 cold-start summary,不可只看 cold-start exit code:
|
||
|
||
- cold-start `BLOCKED>0`:wrapper 才可判定 `BLOCKED`。
|
||
- cold-start `WARN>0 BLOCKED=0`:wrapper 判定為 `SERVICE` warning,結果最多是 `DEGRADED`;不可放大成 `BLOCKED`。
|
||
- cold-start `WARN=0 BLOCKED=0`:才可進入 `FULL_STACK_GREEN` / `FULL_STACK_GREEN_DR_ESCROW_BLOCKED` 判定。
|
||
|
||
### Step 1 - 主機與 SSH
|
||
|
||
```bash
|
||
for host in 192.168.0.110 192.168.0.120 192.168.0.121 192.168.0.188; do
|
||
ping -c 1 -W 1 "$host" >/dev/null && echo "PING_OK $host" || echo "PING_FAIL $host"
|
||
nc -z -w 2 "$host" 22 && echo "SSH_PORT_OK $host" || echo "SSH_PORT_FAIL $host"
|
||
done
|
||
```
|
||
|
||
若任一 P0 host 失敗,不要跳去修 Nginx。先判斷是 power / NIC / fsck / SSH trust / host boot 問題。
|
||
|
||
### Step 2 - 全棧 cold-start scorecard
|
||
|
||
```bash
|
||
scripts/reboot-recovery/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1
|
||
```
|
||
|
||
判定:
|
||
|
||
- `PASS>0 WARN=0 BLOCKED=0`:可進入 `FULL_STACK_GREEN` 候選。
|
||
- `WARN>0 BLOCKED=0`:只能宣稱 `SERVICE_AVAILABLE_DEGRADED`,必須列 WARN。
|
||
- `BLOCKED>0`:不可宣稱恢復完成,先處理第一個 blocker。
|
||
|
||
### Step 3 - MOMO 專用 freshness gate
|
||
|
||
```bash
|
||
scripts/reboot-recovery/momo-drive-token-source-recovery-preflight.sh
|
||
```
|
||
|
||
必要欄位:
|
||
|
||
- `MOMO_HEALTH_VERSION`
|
||
- `SCHEDULER_HEALTH`
|
||
- `TOKEN_STAT` / `CONTAINER_TOKEN_STAT` 只看 metadata,不讀 token。
|
||
- `DB_MONTHLY_SYNC`
|
||
- `DB_DAILY_FRESHNESS`
|
||
- `DB_LATEST_DAILY_IMPORT_JOB`
|
||
|
||
`DB_DAILY_FRESHNESS > 2` 或 import job 失敗時,不可宣稱 MOMO 資料已恢復。
|
||
|
||
### Step 4 - StockPlatform freshness gate
|
||
|
||
```bash
|
||
curl -k -sS https://stock.wooo.work/api/v1/system/freshness
|
||
```
|
||
|
||
必要欄位:
|
||
|
||
- `status` 必須是 `ok`,才可宣稱 StockPlatform 資料層恢復。
|
||
- `latest_trading_date` 必須對齊最近交易日。
|
||
- `core.price_daily`、`core.chips_daily`、`core.market_index_daily.tw` 必須是 `ok`。
|
||
- `blockers` 不可有 `core_margin_short_daily_missing`、`ai_recommendations_stale` 或其他資料閘門阻擋。
|
||
|
||
`stock.wooo.work`、`/healthz`、`/api/healthz` 皆為 200 只代表服務活著;`/api/v1/system/freshness` 回 `blocked` 時,不可宣稱 StockPlatform 資料最新。
|
||
|
||
若 freshness blocker 是 `core_margin_short_daily_missing` / `ai_recommendations_stale`,先分層判斷:
|
||
|
||
- route / health 200:只代表 StockPlatform service up。
|
||
- live source 必須和 Gitea `main` 對齊;2026-06-25 20:11 已知恢復基準是 `fb91aa4c6272469d1d26e0820169629eac17d28a`。
|
||
- `scripts/ops/install-production-cron.sh` 參照的 production cron entrypoints 必須全部存在,否則是 source drift,不是官方資料問題。
|
||
- 自然 cron run 必須不再有 `script_exit_127`。2026-06-25 19:56-20:10 已驗證 source remediation / market index / price / margin / chips / AI pipeline entrypoints 可執行。
|
||
- 若 margin ingestion status=0 但 `row_count=0` 並顯示 official pending,AI pipeline blocked 是正確 fail-closed;不可用手動 DB restore 或 fake freshness 補綠。
|
||
|
||
### Step 5 - Backup / offsite / escrow
|
||
|
||
在 110 只讀執行:
|
||
|
||
```bash
|
||
/backup/scripts/backup-status.sh --no-notify --no-refresh
|
||
```
|
||
|
||
必要欄位:
|
||
|
||
- 110 backup fresh / failed count。
|
||
- 188 backup fresh / failed count。
|
||
- `core_blockers=0`。
|
||
- `integrity_stale=0`。
|
||
- `offsite_fresh=1`。
|
||
- `rclone_gdrive_fresh=1`。
|
||
- `escrow_missing` 必須照實回報。
|
||
|
||
`escrow_missing>0` 時,服務可 green,但 DR 不可 green。
|
||
|
||
### Step 6 - Public routes 只作輔助證據
|
||
|
||
```bash
|
||
for url in \
|
||
https://awoooi.wooo.work/ \
|
||
https://awoooi.wooo.work/api/v1/health \
|
||
https://awoooi.wooo.work/zh-TW/iwooos \
|
||
https://vibework.wooo.work/ \
|
||
https://awooogo.wooo.work/ \
|
||
https://2026fifa.wooo.work/ \
|
||
https://agent.wooo.work/ \
|
||
https://mo.wooo.work/ \
|
||
https://mo.wooo.work/health \
|
||
https://stock.wooo.work/ \
|
||
https://stock.wooo.work/healthz \
|
||
https://stock.wooo.work/api/healthz \
|
||
https://bitan.wooo.work/ \
|
||
https://tsenyang.com/ \
|
||
https://www.tsenyang.com/ \
|
||
https://vtuber.wooo.work/ \
|
||
https://gitea.wooo.work/ \
|
||
https://harbor.wooo.work/ \
|
||
https://registry.wooo.work/ \
|
||
https://sentry.wooo.work/ \
|
||
https://signoz.wooo.work/ \
|
||
https://langfuse.wooo.work/ \
|
||
https://aiops.wooo.work/; do
|
||
code="$(curl -k -sS -o /dev/null -w '%{http_code}' "$url" || true)"
|
||
echo "$code $url"
|
||
done
|
||
```
|
||
|
||
Route smoke 必須和 cold-start / DB / backup 一起看;不能單獨當恢復證明。
|
||
|
||
### Step 7 - 110 CPU / runaway process
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.110 'uptime; vmstat 1 5; ps -eo pid,ppid,pgid,stat,pcpu,pmem,comm,args --sort=-pcpu | head -25'
|
||
```
|
||
|
||
分類:
|
||
|
||
- orphan Chrome / headless smoke:走 runaway process PlayBook,未批准不得 kill。
|
||
- Gitea Actions / CI build / test:先標註短期 CI load,不當事故處理。
|
||
- Docker / DB / Harbor / Sentry 持續高載:回到服務相依與 exporter readback。
|
||
|
||
---
|
||
|
||
## 4. 放行與阻擋口徑
|
||
|
||
| 結果 | 口徑 |
|
||
|------|------|
|
||
| `FULL_STACK_GREEN_DR_ESCROW_BLOCKED` | 可宣稱所有服務面恢復;不可宣稱 DR complete。 |
|
||
| `SERVICE_AVAILABLE_DEGRADED` | 可宣稱服務可用;必須列 WARN 與下一步。 |
|
||
| `BLOCKED_MOMO_DATA_FRESHNESS` | 可宣稱網站可用;不可宣稱資料最新。 |
|
||
| `BLOCKED_STOCK_DATA_FRESHNESS` | 可宣稱 StockPlatform route / container 可用;不可宣稱 StockPlatform 資料或 AI 推薦已最新。 |
|
||
| `BLOCKED_HOST_OR_K3S` | 不可宣稱全棧恢復;先修主機 / K3s。 |
|
||
| `BLOCKED_BACKUP_CORE` | 不可宣稱恢復完成;備份紅燈優先。 |
|
||
| `BLOCKED_WAZUH_REGISTRY` | 不屬於本 SOP 的服務恢復 blocker;必須交給 IwoooS / Wazuh lane,不可改 Wazuh runtime。 |
|
||
| `GREEN_WITH_EVIDENCE_WARNINGS` | 服務可宣稱恢復,但仍有非服務面證據提醒,必須列入 LOGBOOK。 |
|
||
|
||
Wrapper exit code:
|
||
|
||
- `0`:沒有 service blocker。可能仍有 DR boundary / evidence warning。
|
||
- `1`:有 service warning,只能宣稱 degraded。
|
||
- `2`:有 service blocker,不可宣稱恢復完成。
|
||
|
||
---
|
||
|
||
## 5. 完成後必填 LOGBOOK 摘要
|
||
|
||
```text
|
||
時間:
|
||
命令類型:read-only / docs-only / write-with-approval
|
||
主機:110 / 120 / 121 / 188
|
||
Cold-start:PASS=? WARN=? BLOCKED=? RESULT=?
|
||
MOMO:version=? daily_freshness=? latest_job=?
|
||
Backup:110=? 188=? core_blockers=? offsite=? escrow_missing=?
|
||
Routes:列出主要 route code
|
||
CPU / runaway:orphan=? active_ci=? load=?
|
||
仍 blocked:
|
||
不可宣稱:
|
||
```
|
||
|
||
---
|
||
|
||
## 6. 目前最新已驗證基線
|
||
|
||
2026-06-25 21:14 wrapper live run after StockPlatform natural-cron catch-up:
|
||
|
||
- Command type:read-only scorecard / logs / freshness endpoint. 本段無新增 runtime write;20:24 的 targeted `SIGTERM` 已另列於歷史證據。
|
||
- CPU state:110 目前主要負載為 AWOOOI Web `next build` 與其 worker processes;不是 orphan Chrome recurrence。未重啟 Docker、systemd、Nginx、K3s,未取消 CI,未寫 DB。
|
||
- Wrapper:`POST_START_QUICK_CHECK PASS=38 WARN=2 BLOCKED=0`。
|
||
- Warning split:`SERVICE=0 BOUNDARY=1 EVIDENCE=1`。
|
||
- Result:`FULL_STACK_GREEN_DR_ESCROW_BLOCKED`;host / route / K3s / MOMO / StockPlatform / backup service-data gates are green, but DR is not complete.
|
||
- Cold-start:`PASS=89 WARN=0 BLOCKED=0`,Result `GREEN`。
|
||
- MOMO:`V10.690`,dedicated preflight `PASS=19 WARN=2 BLOCKED=0`,job `57` clean,`DB_DAILY_FRESHNESS 1|2026-06-24`。
|
||
- Backup:110 `13/13 fresh failed=0`,188 `2/2 fresh failed=0`,`core_blockers=0`,`offsite_fresh=1`,`rclone_gdrive_fresh=1`,`escrow_missing=5`。
|
||
- StockPlatform:live source and Gitea main `fb91aa4c6272469d1d26e0820169629eac17d28a`; cron entrypoints repaired; 21:00 `intelligence-sync` succeeded; 21:10 `ai-recommendation-pipeline` produced `STOCKPLATFORM_AI_RECOMMENDATION_PIPELINE_OK as_of_date=2026-06-25`; `/api/v1/system/freshness` is `status=ok`, blockers `[]`。Sources: price / chips / margin / AI recommendations all current for `2026-06-25`。
|
||
- Public routes:expanded routes all returned expected `2xx/3xx`。
|
||
- DR:`escrow_missing=5`,不可宣稱 DR complete。
|
||
- Wazuh / SOC:route `200 disabled_waiting_iwooos_wazuh_owner_gate`;manager registry accepted `0`,不是 reboot service blocker。
|
||
|
||
2026-06-25 18:23 wrapper live run after deploy marker `2a9e816a`:
|
||
|
||
- Gitea / CD:`code-review.yaml #3346` success,`cd.yaml #3345` success,deploy marker `2a9e816a chore(cd): deploy aa70835 [skip ci]`。
|
||
- GitOps / K3s:ArgoCD `awoooi-prod` `Synced / Healthy`;API/Web/Worker live image tag `aa70835c7177475430479d8ab68621f59ebeb9b0`;API/Web/Worker ready。
|
||
- Wrapper:`POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`。
|
||
- Warning split:`SERVICE=0 BOUNDARY=1 EVIDENCE=2`。
|
||
- Result:`FULL_STACK_GREEN_DR_ESCROW_BLOCKED`,exit code `0`。
|
||
- Cold-start:`PASS=89 WARN=0 BLOCKED=0`,Result `GREEN`。
|
||
- MOMO:`V10.690`,dedicated preflight `PASS=19 WARN=2 BLOCKED=0`,job `57` clean,`DB_DAILY_FRESHNESS 1|2026-06-24`。
|
||
- Backup:110 `13/13 fresh failed=0`,188 `2/2 fresh failed=0`,`core_blockers=0`。
|
||
- Routes:AWOOI API / IwoooS / Governance / MOMO health / Stock all `200` in direct smoke.
|
||
- Wazuh / SOC:`/api/iwooos/wazuh` and `/api/v1/iwooos/wazuh` route boundary is `200 disabled_waiting_iwooos_wazuh_owner_gate`; `configured=false`, manager registry accepted `0`, runtime gate `0`. Treat this as a security registry evidence blocker, not a reboot service blocker.
|
||
- DR:`escrow_missing=5`,不可宣稱 DR complete。
|
||
- CPU:110 load still elevated around `15.78 / 11.19 / 9.02`; current dominant load is StockPlatform `next build`, StockPlatform headless Chrome smoke, and platform services, not an AWOOI service blocker. Do not kill smoke / CI without explicit approval.
|
||
- Deploy storm note:`d2caa4eb` and `d52583d9` CD runs were superseded by later main commits; use the latest deploy marker and live image readback, not canceled runs, as production truth.
|
||
|
||
2026-06-25 15:04 wrapper live run:
|
||
|
||
- Wrapper:`POST_START_QUICK_CHECK PASS=18 WARN=3 BLOCKED=0`。
|
||
- Warning split:`SERVICE=0 BOUNDARY=1 EVIDENCE=2`。
|
||
- Result:`FULL_STACK_GREEN_DR_ESCROW_BLOCKED`,exit code `0`。
|
||
- Cold-start:`PASS=89 WARN=0 BLOCKED=0`,Result `GREEN`。
|
||
- MOMO:`V10.681`,dedicated preflight `PASS=19 WARN=2 BLOCKED=0`,job `57` clean,`DB_DAILY_FRESHNESS 1|2026-06-24`。
|
||
- Backup:110 `13/13 fresh failed=0`,188 `2/2 fresh failed=0`,`core_blockers=0`。
|
||
- DR:`escrow_missing=5`,不可宣稱 DR complete。
|
||
- CPU:110 顯示 active Gitea Actions / CD `next build` / Playwright e2e smoke;Chrome 是 Playwright child,不是 orphan Chrome。
|
||
|
||
2026-06-25 14:41 wrapper live run:
|
||
|
||
- Wrapper:`POST_START_QUICK_CHECK PASS=18 WARN=2 BLOCKED=0`。
|
||
- Warning split:`SERVICE=0 BOUNDARY=1 EVIDENCE=1`。
|
||
- Result:`FULL_STACK_GREEN_DR_ESCROW_BLOCKED`,exit code `0`。
|
||
- Cold-start:`PASS=89 WARN=0 BLOCKED=0`,Result `GREEN`。
|
||
- MOMO:`V10.676`,dedicated preflight `PASS=19 WARN=2 BLOCKED=0`,job `57` clean,`DB_DAILY_FRESHNESS 1|2026-06-24`。
|
||
- Backup:110 `13/13 fresh failed=0`,188 `2/2 fresh failed=0`,`core_blockers=0`。
|
||
- DR:`escrow_missing=5`,不可宣稱 DR complete。
|
||
- CPU:110 顯示 active CI / build / test load;沒有 orphan Chrome 復發證據。
|
||
|
||
2026-06-25 14:16:
|
||
|
||
- Cold-start:`PASS=89 WARN=0 BLOCKED=0`,Result `GREEN`。
|
||
- MOMO:`V10.674`,job `57` clean,`DB_DAILY_FRESHNESS 1|2026-06-24`。
|
||
- Backup:110 `13/13 fresh failed=0`,188 `2/2 fresh failed=0`,`core_blockers=0`。
|
||
- DR:`escrow_missing=5`,不可宣稱 DR complete。
|
||
- Wazuh:host registry accepted 仍不屬於本 SOP 完成項,不可宣稱全部主機納管完成。
|