# AWOOOI 全棧冷啟動與主機重啟 SOP > Version: v1.131 > Last updated: 2026-07-09 Asia/Taipei > Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號，但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal；此納入不代表授權任何破壞性 runtime apply。 --- ## 0. 最新 Live Baseline 與釋出判定 本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。 v1.131 AI log triage controlled auto-recovery rule：每次全主機重啟後，10 分鐘 SLO verifier 不得只跑 read-only summary。`awoooi-reboot-auto-recovery-slo.service` 必須以 `AI_LOG_TRIAGE_AUTO_RECOVERY_MODE=apply` 先執行 `scripts/reboot-recovery/ai-log-triage-auto-recovery.sh`，bounded 收集 99 / 110 / 120 / 121 / 188 logs、K3s events、VMware no-secret readback 與 public route status，自動輸出 findings / actions / summary，再接 host probe、post-reboot summary、scorecard 與 textfile metrics。2026-07-09 16:23 orchestrator check artifact `/tmp/awoooi-full-host-reboot-orchestrator-codex/20260709T082305Z` 的目前 truth 是 `RECOVERY_BLOCKED_P0`、`FINDING_COUNT=3`、`FINDING_P0_COUNT=2`：`STOCK_UPSTREAM_NOT_READY host=110_188`、`EDGE_502_UPSTREAM_REFUSED host=188_edge`，另有 `HOST188_POSTGRES_SYSTEMD_FAILED` P2 hygiene；public route readback 為 AWOOOI health `200`、AWOOOI web `307`、Stock freshness `200`、Gitea `200`、registry `/v2/` `401`、Harbor `200`。這些不得再被當成噪音或人工 SOP；必須進 AI findings、Prometheus/Telegram blocker 與下一步 action matrix。允許的 `--apply` 只限低風險 allowlist：start inactive docker/nginx、Harbor watchdog log-first repair、registry ready 後刪 failed-pull pods 讓 K3s 重建。仍禁止 reboot、node drain、firewall cutover、DB destructive action、Docker prune、secret/session/.env/raw-volume reads。 v1.130 Windows99 VMX backup/source search package rule：2026-07-03 18:19 最新 via-110 no-secret locator 已把目前 truth 從早先 `candidate_source_count=6` 修正為 `candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`，且 `raw_path_output=false`、`path_fingerprint_only=true`、`secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`。因此 P0-4 下一步不得再引用 generic Ubuntu candidate；必須讀 `/api/v1/agents/windows99-vmx-source-backup-search-package` 與 priority readback 的 `windows99_vmx_source_backup_search_*` 欄位。當前 backup/source search receipt 顯示 120 / 121 / local 均未找到 verified 111 source，99 alias 無 Linux backup host mapping，110 / 188 / 112 SSH BatchMode 不可用，`candidate_source_count=0`、`blocked_host_count=4`、`apply_allowed_by_this_package=false`。固定下一步是離線備份 inventory 或授權 console source path，再取得 verified source path 後建立 scoped relink dry-run；仍不得 power on VM、restore、讀 VMX 內容、讀 secret、重啟 Windows / VMware service / host / Docker / Nginx / K3s / DB / firewall。 若只是重啟後要快速判斷能不能宣稱恢復，先跑機器可讀摘要：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color`。此腳本會呼叫一頁式總檢查、188 host hygiene checklist 與 Wazuh no-false-green repo gates，並把 delegated logs 和可重放的 `summary.txt` 留在 `/tmp/awoooi-post-reboot-readiness-*`。v1.75 起，同一輪驗收後續步驟必須吃同一個 `$ARTIFACT_DIR/summary.txt`，例如 `scripts/reboot-recovery/post-reboot-declaration-guard.py --summary-file "$ARTIFACT_DIR/summary.txt" --no-color` 與 `scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --summary-file "$ARTIFACT_DIR/summary.txt" --no-color`；不得在同一輪 evidence chain 反覆重跑 live probes 後混用不同時間點結論。v1.76 起，delegated cold-start 若在 K3s rollout / CD 替換瞬間出現單次 `BLOCKED AWOOOI API not reachable`，但 wrapper 自己的 public `https://awoooi.wooo.work/api/v1/health` route retry 已回 2xx，該 blocker 只列為 route/API warmup evidence warning；public API 仍失敗、其他 non-route blocker、或 retry 後未恢復時，仍維持 hard blocked。宣告 guard 會把 summary 轉成 allowed / forbidden declaration，避免把服務綠誤報成 DR complete、188 host hygiene、Wazuh registry recovered 或 runtime authorized。若 summary 顯示 `SERVICE_GREEN=1` 但 `NEXT_REQUIRED_GATES` 仍非空，再由 dispatch checklist 把尚未完成的 blocker 轉成 owner / evidence / forbidden-action checklist；需要機器可讀 intake 時，再跑 `scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --summary-file "$ARTIFACT_DIR/summary.txt" --output /tmp/awoooi-post-reboot-owner-packets.json` 產生 `awoooi_post_reboot_next_gate_owner_packets_v1` JSON，立刻跑 `scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json`，並用 `scripts/reboot-recovery/post-reboot-owner-response-template.py --owner-packet-file /tmp/awoooi-post-reboot-owner-packets.json --output /tmp/awoooi-post-reboot-owner-response-template.json` 產生只含 active gates 的 placeholder response。dispatch / packet / guard / template 均固定 `DISPATCH_AUTHORIZED=0`、`REQUEST_SENT_COUNT=0`、`OWNER_RESPONSE_ACCEPTED=0`、`HOST_WRITE_AUTHORIZED=0`、`SECRET_VALUE_COLLECTION_ALLOWED=0`、`RUNTIME_GATE=0`；guard 未通過時不得送 owner request、不得寫 escrow marker、不得進維護窗口、不得宣稱 DR / Wazuh registry complete。v1.74 起，任何 owner response JSON 還必須經過 `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --owner-packet-file --response-file `：空模板、placeholder、secret payload、runtime action request、credential marker write、Wazuh active response / re-enroll / restart、Kali active scan 或缺少 active gate evidence 都必須 fail-closed；preflight 通過也只表示可進入獨立 reviewer acceptance，不是 runtime 授權。需要人工展開時，再跑 `scripts/reboot-recovery/post-start-quick-check.sh --no-color` 並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。 v1.76 owner gate replay rule：同一輪 summary 產生後，owner packet 與 owner response preflight 必須優先使用 `--summary-file "$ARTIFACT_DIR/summary.txt"`，例如 `scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color --summary-file "$ARTIFACT_DIR/summary.txt" --output /tmp/awoooi-post-reboot-owner-packets.json` 與 `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --summary-file "$ARTIFACT_DIR/summary.txt" --response-file `。只有在刻意要重新取 live evidence 時，才允許省略 `--summary-file`；否則 preflight 不得自己重跑 summary 造成同一輪狀態漂移。 v1.79 active owner response template rule：同一輪 owner packet 產生後，placeholder response 必須由 `scripts/reboot-recovery/post-reboot-owner-response-template.py --owner-packet-file ` 生成，讓 `responses[].gate_id` 等於 active `owner_packets[].packet_id`。目前 2026-06-29 09:13 readback 只剩 `credential_escrow_evidence`，因此 generated template 不得帶入 `wazuh_manager_registry_export`。placeholder template 必須被 preflight 擋在 `blocked_waiting_owner_response_content`、`received=0`、`accepted=0`、`runtime_gate=0`；它是 no-secret intake aid，不是 owner accepted 或 marker-write 授權。 v1.80 / v1.81 credential escrow intake scorecard rule：同一輪 owner response preflight 後，必須用 `scripts/reboot-recovery/post-reboot-credential-escrow-intake-scorecard.py --summary-file "$ARTIFACT_DIR/summary.txt" --owner-packet-file --response-file --offsite-report-file --escrow-status-file ` 收斂 DR escrow gate。scorecard 只讀 sanitized artifacts；不得讀 secret value、不得寫 marker、不得送 owner request、不得開 runtime gate。placeholder readback 期望 `STATUS=blocked_waiting_non_secret_credential_escrow_evidence`、`EFFECTIVE_ESCROW_MISSING_COUNT=5`、`OWNER_RESPONSE_RECEIVED_COUNT=0`、`OWNER_RESPONSE_ACCEPTED_COUNT=0`、`RUNTIME_GATE_COUNT=0`、`CREDENTIAL_MARKER_WRITE_AUTHORIZED_COUNT=0`。若未來收到合格 redacted owner response 並由 preflight 回 `ready_for_independent_reviewer_acceptance`，scorecard 應轉為 `STATUS=ready_for_independent_reviewer_acceptance`；即使 marker 尚未寫入，也只能進 `independent_reviewer_acceptance_then_marker_dry_run`，不得直接寫 marker 或宣稱 `DR_COMPLETE`。 v1.93 reboot SLA readback fixed fields rule：`scripts/reboot-recovery/reboot-auto-recovery-slo-scorecard.py`、`docs/operations/awoooi-reboot-auto-recovery-slo-scorecard.snapshot.json` 與 `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 必須固定輸出 `current_phase`、`eta_or_wait_reason`、`primary_blocker`、`active_blockers`、`next_safe_action` 與 `reboot_sop_progress.fixed_triage_order`。回報時先讀這些欄位，不得重新臨場發明排查順序；若 `current_phase=host_boot_detection_blocked` 且 `eta_or_wait_reason=target_window_elapsed_eta_unavailable_until_fresh_all_host_reboot_event_and_probe`，只能宣稱「10 分鐘 SLO 尚未證明」，下一步固定補 fresh all-host reboot event / probe 與 99 VMware/Windows readback，不得把部分 Linux host green 當全主機 green。 v1.94 Windows 99 / VMware verifier rule：99 主機與 guest VM 自動啟動不得再只靠「ping 到 99」或「VMX source 存在」宣稱。`scripts/reboot-recovery/windows99-vmware-autostart.ps1 -Mode Verify` 必須輸出 no-secret 固定欄位：`VMRUN_PRESENT`、`VMX alias=... present=...`、`VMWARE_SERVICE ... ok=...`、`VMWARE_AUTOSTART_TASK ... ok=...`、`WINDOWS_UPDATE_POLICY ... ok=...`、`VM_POWER alias=... running=...`、`VMWARE_AUTOSTART_CONFIG_READY`、`VMWARE_AUTOSTART_POWER_READY`、`WINDOWS_UPDATE_NO_AUTO_REBOOT_READY`、`VMWARE_AUTOSTART_VERIFY_READY`。`reboot-auto-recovery-slo-scorecard.py --windows99-vmware-file ` 必須解析這些欄位；缺 readback 時 active blocker 固定為 `windows99_vmware_autostart_readback_missing`，next action 固定為 `collect_windows99_vmware_autostart_verify_readback_then_rerun_all_host_reboot_scorecard_no_secret_no_reboot`。預設 required guest VM alias 以本輪 reboot 事故 source-of-truth 為準：`111 / 188 / 120 / 121 / 112`；`110` 保留為 required host / control-plane reachability，不得被當成 99 VMware 預設 VM。此 verifier 不讀 Windows 密碼、不讀 secret、不啟動 VM、不重啟 host；`Apply` 仍需獨立 controlled apply 與 post-verifier。 v1.95 Windows 99 management-channel readback rule：若 99 可 ping / RDP / TCP，但 `windows99-vmware-autostart.ps1 -Mode Verify` 尚未收回，不得只寫成「等 verifier」。必須先跑 `scripts/reboot-recovery/windows99-management-channel-probe.py --output `，並把結果用 `reboot-auto-recovery-slo-scorecard.py --windows99-management-file ` 接入同一份 SLO scorecard。此 probe 只做 no-secret readback：TCP `22 / 135 / 445 / 2179 / 3389 / 5985 / 5986`、SSH BatchMode publickey、WinRM port、RDP console reachability 與 VMConnect / local console reachability；不得讀 Windows 密碼、不得啟動 VM、不得重啟、不得改 Windows Update。2026-07-02 18:28 live readback：`host_reachable=true`、`rdp_console_reachable=true`、`hyperv_vmconnect_open=true`、`local_console_channel_reachable=true`、`console_collection_channels=["rdp_console","hyperv_vmconnect"]`、`ssh_batch.status=permission_denied`、`winrm_http_open=false`、`winrm_https_open=false`、`remote_execution_channel_ready=false`；因此 active blockers 必須包含 `windows99_remote_execution_channel_unavailable` 與 `windows99_vmware_autostart_readback_missing`，next action 固定為 `restore_windows99_no_secret_management_channel_or_collect_local_console_verify_readback_then_rerun_reboot_scorecard_no_reboot`。 v1.96 reboot SLO per-blocker alert projection rule：重啟後不得只看 `awoooi_reboot_auto_recovery_slo_blocker_count` 或 `RebootAutoRecoverySLOMissed` 總告警。`scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh` 必須從同一份 `scorecard.json` 產生 `awoooi_reboot_auto_recovery_slo_active_blocker{blocker=...}` 與 `awoooi_reboot_auto_recovery_slo_primary_blocker{blocker=...}` textfile metrics；Prometheus 必須載入 `RebootAutoRecoveryActiveBlocker` 與 `RebootAutoRecoveryActiveBlockerMetricMissing`。Telegram / Alertmanager 看到 blocker label 後，先按 `blocker` 指向的固定 lane 處理：`windows99_*` 走 no-secret collector 或 console verify、`backup_*` 走 backup-status / exporter readback、`wazuh_dashboard_degraded` 走 Wazuh registry/API 只讀、`host_*` / `reboot_event_*` 走 host-probe / reboot-event-state，同時禁止 reboot、Docker / systemd / Nginx / DB / firewall restart、restore、prune、delete。若 blocker_count 大於 0 但 active_blocker metric 缺失，視為監控鏈缺口，不可宣稱「有告警」已足夠。 v1.97 reboot SLO action-matrix routing rule：重啟後不得只把 active blocker 丟給人工判讀。`reboot-auto-recovery-slo-scorecard.py`、production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 與 exporter artifact 必須輸出 `active_blocker_action_matrix`，每個 blocker 固定含 `category`、`owner_lane`、`telegram_severity`、`evidence_inputs`、`next_safe_action`、`post_verifier`、`controlled_apply_mode` 與 `forbidden_actions`。`awoooi_reboot_auto_recovery_slo_active_blocker` metric 必須帶 `category`、`owner_lane`、`severity`、`primary` labels，Telegram / Alertmanager 優先讀這些 labels；若 labels 缺失，只能判為 exporter/action-matrix drift。`windows99_remote_execution_channel_unavailable` 必須排在 `windows99_vmware_autostart_readback_missing` 前，因為未恢復 no-secret collection channel 時不可只等 VMware verifier。所有 action matrix row 的 `controlled_apply_authorized_by_scorecard=false`，表示 scorecard 只授權 verifier / check-mode 路由，不授權重啟、VM power change、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete 或 secret 讀取；低風險 controlled apply 必須進各自 lane 的 check-mode、rollback、post-verifier。 v1.98 reboot-event detector unknown-uptime rule：`reboot-event-detector.py` 不得把 `uptime_seconds=unknown`、`boot_id=unknown` 或 `boot_id=reachable_unknown_boot` 當作 fresh reboot 或 boot id changed。只有 `reachable=true` 且 `uptime_seconds >= 0` 且 `uptime_seconds <= target_seconds` 才能進 `fresh_boot_hosts`；只有前後兩個 boot id 都不是 placeholder 時才能判定 `boot_id_changed`。若 99 只剩 ping / TCP reachable、111 unreachable、188 degraded 或 startup failed，必須維持 `all_required_hosts_in_reboot_window=false` 與 SLO blocked；不得用 ping-only host 當 10 分鐘內重啟證據。 v1.99 runtime blocker action-matrix classification rule：Prometheus / runtime overlay 帶回的 `conversation_event_hot_path_index_migration_source_missing` 不得留在 `reboot_slo_unknown`。此 blocker 屬於 `host_cpu_pressure`，owner lane 固定為 `host_pressure_controller`，next action 固定為 `restore_conversation_event_hot_path_migration_source_then_rerun_host_pressure_and_reboot_slo_scorecard_no_restart`；evidence inputs 必須包含 `source_controls` 與 `runtime_metric_readback`。此規則只授權 source-control / verifier 修復與 check-mode，仍不得直接做 DB destructive migration、kill process、Docker / DB / K3s / Nginx restart 或 reboot。 v1.100 host-probe bounded TCP verifier rule：`scripts/reboot-recovery/reboot-auto-recovery-host-probe.sh` 的 reachable-only TCP fallback 不得只依賴 `nc -w`；每一個 `nc -z` port probe 必須再包一層 `run_with_timeout "${TCP_CONNECT_TIMEOUT_SECONDS:-2}"`，避免單台主機或單個 port 卡住後讓後續主機缺列，造成 10 分鐘 SLO readback 無法判斷。若 verifier timeout 或主機缺列，`reboot-event-detector.py` / `reboot-auto-recovery-slo-scorecard.py` 必須 fail-closed 成 host boot detection blocker；不得用前半段 partial probe 宣稱 all-host observed、fresh reboot 或 auto-recovery ready。2026-07-03 live no-write 驗證 artifact `/tmp/awoooi-reboot-host-probe-confirm-20260703-015844`：bounded host probe 約 `10.825s` 完成、`HOST_BOOT` rows `7`、`missing_hosts=[]`；但 111 仍 unreachable、99 仍 `uptime_seconds=unknown`、188 仍 `systemd_state=degraded` / `startup_active=failed`，因此 10 分鐘全主機自動恢復 SLO 仍維持 blocked。 v1.101 reboot SLO exporter timeout fallback rule：`scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh` 不得在 `post-reboot-readiness-summary.sh` timeout 時只寫 `POST_REBOOT_READINESS_SUMMARY_TIMEOUT=1` 而抹掉已完成的 `post-start-quick-check.log` 綠燈證據。timeout 預設為 `POST_REBOOT_READINESS_TIMEOUT_SECONDS=120`；若 full summary timeout 但 post-start log 已有摘要，exporter 必須寫出 partial `summary.txt` 並包含 `POST_REBOOT_READINESS_PARTIAL_FROM_POST_START=1`、`SERVICE_GREEN`、`PRODUCT_DATA_GREEN`、`BACKUP_CORE_GREEN`、`POST_START_RESULT`、`POST_START_BLOCKED` 等欄位。scorecard 不得只因 timeout artifact 就把 service / product data / backup core 重開成 active blocker；尚未讀回的 188 hygiene、Wazuh、DR 或 Windows99 欄位必須標成 partial / unknown，由各自 blocker lane 處理。 v1.102 production scorecard / deploy cutover rule：2026-07-03 03:55 production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 的目前事實是 `active_blocker_count=8`、`readiness_percent=67`、`can_claim_all_services_recovered_within_target=false`；`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`HOST_188_SERVICE_GREEN=1`、`WAZUH_DASHBOARD_DEGRADED=false`、Gitea bundle backup `expected_repo_count=12` / `repo_row_count=12` / `all_expected_ok=true`，StockPlatform freshness `status=ok` 且 `latest_trading_date=2026-07-02`。因此 service/data/backup/Wazuh/Gitea bundle blocker 不得因舊 Prometheus label 或 summary timeout 被重開；剩餘 P0 blocker 固定為 host boot detection / all-host fresh reboot event、111 unreachable、99 uptime unknown、Windows99 VMware VMX / guest power verifier。另，deploy marker 切換期間若 public API / SLO route 出現 502 且沒有 maintenance body / fallback header，即使稍後自癒，也必須列入 P0 public UX blocker；下一步固定是 source-controlled deploy drain、maintenance fallback verifier 與 public route watch，不得把短暫 502 當作正常噪音。 v1.103 Windows99 console verify artifact rule：若 99 可 ping / RDP / Hyper-V VMConnect，但 SSH BatchMode 與 WinRM 都不可用，不能停在「等人工」。必須把 console channel 的 `windows99-vmware-autostart.ps1 -Mode Verify` stdout 存成 no-secret artifact，先跑 `python3.11 scripts/reboot-recovery/validate-windows99-console-verify-artifact.py --input --output `，再用 normalized artifact 重跑 reboot SLO scorecard。validator 必須檢查 `AWOOOI_WINDOWS99_VMWARE_AUTOSTART=1`、`MODE=Verify`、每個 required alias 的 `VMX` / `VM_POWER`、VMware services、autostart task、Windows Update policy 與 ready flags；若看到 secret-like key、欄位缺失或非 Verify marker 必須 fail-closed。此路徑只接受 Verify stdout，不授權 Windows password collection、VM power change、Windows Update policy apply、service restart、host reboot 或 registry edit。 v1.104 public maintenance location-level intercept rule：deploy marker 切換或主機重啟期間，public API / SLO route 只要回 `502` 且沒有 `X-AWOOOI-Fallback=local-maintenance` 與維護頁 body，即使稍後恢復，也必須列為 P0 public UX blocker。`ops/nginx/awoooi.wooo.work.conf`、`k8s/nginx/awoooi-prod.conf` 與 `ops/maintenance/nginx-502-maintenance-snippet.conf` 必須使用 `error_page 502 503 504 =503 /__awoooi-maintenance.html;`，讓 fallback 明確回 `503` / `Retry-After`；所有 proxy 到 AWOOOI API / Web 的 `location` 必須各自宣告 `proxy_intercept_errors on;` 與同一條 `error_page`，不得只依賴 server 層繼承。`public-maintenance-fallback-probe.py` 需將帶 fallback header/body 的 `502` / `503` / `504` 判定為 maintenance fallback serving；只有缺 header/body 的 5xx 才算 `raw_5xx_without_maintenance_fallback`。完成條件仍是 marker-time public route watch 不再讀到 raw 502；source contract 通過不等於 live Nginx runtime 已套用。 v1.105 public maintenance edge source-to-live drift rule：Gitea CD `#4519` 已部署 `a94ddd5` 並推 deploy marker `3aca484`，但 marker 後 public probe 仍讀到 `https://awoooi.wooo.work/api/v1/health` raw `502`、無 fallback header/body；稍後 route 自癒為 `200` 不得視為完成。live readback 證實 188 使用 `/etc/nginx/sites-enabled/awoooi.wooo.work.conf`，該檔仍是 VIP `192.168.0.125` 版舊配置但缺 `proxy_intercept_errors` / `error_page` / `X-AWOOOI-Fallback`，且 `/var/www/maintenance/maintenance.html` 缺失。repo `ops/nginx/awoooi.wooo.work.conf` 必須保留 VIP `192.168.0.125:32334/32335`，不得用舊 `192.168.0.120` config 覆蓋 live。`scripts/reboot-recovery/public-maintenance-edge-fallback-apply.sh --check` 是 no-secret readback；`--apply` 只能在 passwordless sudo channel 存在時執行，必須備份 live conf、安裝 maintenance file、`nginx -t` 通過、reload、再 rerun public route probe。若 `SUDO_N_READY=0`，active blocker 固定為 `privileged_edge_apply_channel_unavailable`，不得讀密碼、不得重啟 Nginx / Docker / host，也不得宣稱 P0-2 完成。 v1.107 2026-07-03 08:23 reboot verify-only readback rule：production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 仍回 `status=blocked_reboot_auto_recovery_slo_not_ready`、`active_blocker_count=8`、`readiness_percent=67`、`primary_blocker=reboot_event_required_host_unreachable`，active blockers 固定為 `all_required_hosts_not_in_10_minute_reboot_window`、`fresh_all_host_reboot_event_missing`、`host_boot_observation_older_than_target_window`、`host_unreachable_after_reboot`、`host_uptime_unknown`、`reboot_event_required_host_unreachable`、`windows99_vmware_guest_power_not_ready`、`windows99_vmware_vmx_missing`。同輪 no-write artifact `/tmp/awoooi-reboot-verify-only-20260703-082310` 證明：99 ping/RDP/Hyper-V VMConnect 可達但 uptime unknown，SSH BatchMode / WinRM 不可用；111 仍 unreachable；112/120/121 可達但 uptime 已超過 10 分鐘目標窗口；188 可達但 `systemd_state=degraded`、`awoooi-startup.service failed`。因此不得宣稱 10 分鐘 SLO 已完成，也不得把 RDP 可見或 source verifier 存在當作 VMware autostart ready。下一步固定為 `restore_windows99_no_secret_management_channel_or_collect_local_console_verify_readback_then_rerun_reboot_scorecard_no_reboot`，並同時保持 `rerun_reboot_event_detector_and_host_probe_verify_only_no_reboot`；禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。 v1.108 Windows99 console artifact reliability blocker rule：若 RDP / Hyper-V VMConnect 畫面可見，但 clipboard paste、視窗 focus 或 stdout 擷取不可靠，不能把畫面片段、截斷 output 或 Windows 端舊剪貼簿內容當成 `windows99-vmware-autostart.ps1 -Mode Verify` artifact。`windows99-management-channel-probe.py` 必須輸出 `console_artifact_status`、`console_artifact_reliable`、`console_artifact_blockers` 與 `console_artifact_safe_next_step`；`blocked_clipboard_unreliable`、`blocked_focus_unreliable`、`blocked_truncated_output` 分別轉成 `windows99_console_clipboard_unreliable`、`windows99_console_focus_unreliable`、`windows99_console_verify_output_truncated`。這些 blocker 必須進 `active_blockers`、`windows99_verify_collection.collection_blockers`、readback、rollups 與 `active_blocker_action_matrix`，owner lane 固定為 `windows99_console_or_no_secret_management_channel`，next safe action 固定為 `stop_unreliable_rdp_clipboard_path_and_use_authorized_no_secret_management_channel_or_validated_console_stdout_artifact`。完成條件仍是取得 validator 可接受的完整 no-secret Verify stdout 或恢復授權 no-secret management channel；不得讀 Windows 密碼、不得啟動 / 關閉 VM、不得重啟 host / service。 v1.109 Windows99 no-secret collector candidate coverage rule：`collect-windows99-vmware-verify.sh --check` 預設必須 probe 全部已列 SSH candidate，而不是只 probe 前兩個帳號；`--max-auth-users N` / `WINDOWS99_MAX_AUTH_USERS` 只作為明確縮小 blast radius 的 override。所有 probe 仍必須固定 `BatchMode=yes`、`PreferredAuthentications=publickey`、`PasswordAuthentication=no`、`KbdInteractiveAuthentication=no`、`NumberOfPasswordPrompts=0` 與 bounded timeout；不得用密碼 prompt、sshpass、WinRM credential、遠端寫入、VM power change 或 Windows policy apply 來補 verifier。若全候選仍回 `ssh_batchmode_auth_ready=0`，active blocker 必須保持 `windows99_ssh_publickey_auth_missing` / `windows99_remote_execution_channel_unavailable`，下一步才是恢復授權 no-secret management channel 或取得 validator 可接受的 console Verify stdout artifact。 v1.110 Windows99 nested runtime blocker promotion rule：production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 不得只信 Prometheus `active_blocker` label 而忽略 Windows99 verifier 巢狀 evidence。若 `windows99_vmware_autostart.service_blockers`、`autostart_task.ok=false` 或 `policy_blockers` 存在，API overlay 必須把 `windows99_vmware_autostart_config_not_ready` 與 `windows99_update_no_auto_reboot_policy_not_ready` 重新提升到 `active_blockers`、`active_blocker_action_matrix`、readback 與 rollups，並列入 `runtime_metric_runtime_readback_added_blockers`。這些 blocker 只授權 no-secret verifier、source config / check-mode package 與 post-verifier；不得從 scorecard 直接重啟 Windows service、改 registry、套 Windows Update policy、啟動 / 關閉 VM 或重啟 host。 v1.111 Gitea CD Docker step bounded-timeout rule：public queue 顯示 `cd.yaml` Running 不得視為 production 已更新或中性等待；必須同時讀 `latest_visible_cd_deploy_marker_*`、`latest_visible_cd_production_deploy_readback_matched_from_log`、`latest_visible_cd_failure_classifier`、`latest_visible_cd_inflight_classifier` 與 `latest_visible_cd_docker_step_timeout*`。`Build and Push API` / `Build and Push Web` 的 `docker build` 與 `docker push` 必須由 `run_docker_step` 包住，預設 API build `1200s`、Web build `1500s`、push `600s`，timeout 時 fail-closed 輸出 `BLOCKER cd_docker_step_timeout label=... timeout_seconds=...`。`read-public-gitea-actions-queue.py --json` 必須把此 blocker 上卷為 `blocked_cd_docker_step_timeout` 與固定 safe-next-action；下一步只能檢查 runner Docker build/push path、registry route 與 queue readback，不得用 workflow_dispatch、重啟 Docker / host / Nginx / K3s / DB、讀 secrets 或恢復 generic runner 掩蓋卡點。 v1.112 current production SLO readback anchor：2026-07-03 10:50 production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 的目前事實是 `status=blocked_reboot_auto_recovery_slo_not_ready`、`active_blocker_count=10`、`readiness_percent=60`、`can_claim_all_services_recovered_within_target=false`、`primary_blocker=reboot_event_required_host_unreachable`。active blockers 固定為 `all_required_hosts_not_in_10_minute_reboot_window`、`fresh_all_host_reboot_event_missing`、`host_boot_observation_older_than_target_window`、`host_unreachable_after_reboot`、`host_uptime_unknown`、`reboot_event_required_host_unreachable`、`windows99_vmware_guest_power_not_ready`、`windows99_vmware_vmx_missing`、`windows99_vmware_autostart_config_not_ready`、`windows99_update_no_auto_reboot_policy_not_ready`；`runtime_metric_runtime_readback_added_blockers` 固定含 Windows99 config / policy 兩項，`windows99_update_no_auto_reboot_ready=false`。下一步固定為 `restore_windows99_missing_vmx_source_for_aliases_then_rerun_no_secret_collector_and_scorecard_no_vm_power_change`。不得再用 08:23 的 `8 / 67%` 當目前狀態；它只能保留為歷史 readback。此錨點仍只授權 source / verifier / check-mode package，不授權 VM power change、Windows service restart、registry apply、host reboot、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete 或 secret 讀取。 v1.113 Windows99 VMX source repair package API rule：production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 必須直接輸出 `windows99_vmx_source_repair_package`，並在 `readback` / `rollups` 同步 `windows99_vmx_source_repair_package_ready`、`windows99_vmx_source_repair_package_status`、`windows99_vmx_source_repair_package_safe_next_step`、`windows99_vmx_source_repair_package_apply_allowed` 與 `windows99_vmx_source_repair_missing_vmx_aliases`。package 只從已讀回的 Windows99 verifier / Prometheus runtime overlay 建立，Prometheus overlay 後必須重建，避免 committed snapshot 與 live metric 不一致。111 VMX missing 時，package 必須列出 expected path `D:\Documents\Virtual Machines\192.168.0.111_Ubuntu_64-bit\192.168.0.111_Ubuntu_64-bit.vmx` 與 `Test-Path -LiteralPath ...` check-mode probe；`apply_allowed_by_this_package=false` 固定不變。此 package 是 work item / Telegram / UI 的 next-action input，不是 runtime apply 授權；不得從此 package 直接做 VM power change、Windows service restart、registry apply、scheduled task modify、host reboot 或 secret/password 讀取。 v1.114 Windows99 VMX package public redaction boundary rule：public `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 與 `/api/v1/agents/windows99-vmx-source-repair-package` 會先套用 LAN topology redaction，因此 VMX path 內的 `192.168.0.111` 可能在 public JSON 變成 `host:dev-a` 類 display alias。public response 必須同步輸出 `public_lan_topology_redacted=true`、`redacted_public_display_only=true`、`check_mode_probe_commands_executable=false` 與 `internal_console_package_required_for_execution=true`；所有 `vm_rows[]` 也必須帶相同 executable=false 邊界。任何 `host:...` VMX path 只能供瀏覽器 / Telegram / UI 顯示，不得作為 Windows console 可執行命令。要跑 `Test-Path` 時只能使用授權內部 console / no-secret package 保留的原始 Windows path；仍不得讀密碼、啟動 VM、重啟 service / host 或套用 registry。 v1.115 Windows99 VMX source locate check-mode rule：2026-07-03 12:34 deploy 後 production readback 已更新為 `status=blocked_reboot_auto_recovery_slo_not_ready`、`active_blocker_count=10`、`readiness_percent=60`、`primary_blocker=reboot_event_required_host_unreachable`、`can_claim_all_services_recovered_within_target=false`；priority readback 同步顯示 `windows99_vmx_source_repair_action_stage=source_repair_check_mode_ready`、`windows99_vmx_source_repair_next_executable_step=locate_or_relink_missing_vmx_source_check_mode`、`windows99_vmx_source_locator_status=check_mode_locator_ready_windows99_vmx_source_required`，locator endpoint 顯示 `missing_vmx_aliases=["111"]` 與 safe command `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect`。定位 111 VMX source 時固定使用 no-secret locator / collector；本輪 live check-mode 結果：expected 111 VMX path `exists=0`、`candidate_vmx_count=6`、`alias_hint_candidate_count=0`、`identity_target_hint_count=0`、`identity_unassigned_ubuntu_candidate_count=1`、`locate_status=single_identity_unassigned_ubuntu_candidate_requires_confirmation`。該 generic Ubuntu candidate 不能直接 relink；先前 MAC / neighbor evidence 未證明它是 111，且 99 local named backup roots 與 110 bounded backup roots 未找到 111 VMX artifact。下一步只能是取得外部 verified backup artifact 或更強身份證據後建立 scoped relink / restore dry-run，再跑 post-verifier；不得用猜測候選直接 relink、power on 或宣稱 SLA 完成。 v1.116 Windows99 VMX locator via-host execution rule：若目前工作站直連 99 缺 public-key auth，但 110 到 99 的 no-secret SSH path 已驗證，官方 locator 必須支援 `--via-host`，固定可執行命令為 `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect --via-host wooo@192.168.0.110 --users wooo --max-auth-users 1`。2026-07-03 live readback：`locator_collection_status=collected_windows99_vmx_source_locator_stdout`、`expected_vmx_path_present_count=0`、`candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`、`safe_next_step=provide_existing_vmx_source_or_backup_path_then_rerun_locator_check_mode_no_write`，且 `secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`、`raw_path_output=false`、`path_fingerprint_only=true`。此結果把下一步收斂為「提供既有 VMX source 或 verified backup path」；仍不得 relink generic VM、啟動 VM、restore 到 production 或讀密碼。 v1.117 Windows99 VMX candidate confirmation gate rule：locator receipt 顯示 `single_identity_unassigned_ubuntu_candidate_requires_confirmation` 時，不能把「唯一 generic Ubuntu candidate」直接升級成 111 VMX source。`/api/v1/agents/windows99-vmx-source-locator-readback` 必須輸出 `candidate_confirmation_gate`，並同步到 priority readback / Work Items；目前 gate 固定 `candidate_identity_evidence_sufficient=false`、`candidate_relink_dry_run_ready=false`、`apply_allowed_by_candidate_confirmation_gate=false`，missing evidence 至少包含 `target_alias_identity_hint`、`verified_backup_or_authorized_console_candidate_identity`、`pre_apply_backup_plan`、`scoped_relink_dry_run`。只有取得 target alias / MAC / UUID / display-name 或 verified backup path 證據，且產生 rollback + post-verifier 的 scoped relink dry-run 後，才可進下一階段；scorecard / locator / work item 均不得授權 VM power change、Windows write、service restart、host reboot、secret/password 讀取或 production restore。 v1.118 Windows99 VM target source-of-truth reconciliation rule：99 VMware autostart 的預設 target 不得與目前 reboot 事故目標分叉。repo source、API scorecard fallback、CLI scorecard fallback、validator default 必須一致使用 `111 / 188 / 120 / 121 / 112`；`110` 仍是全主機 reboot SLO 的 required host 與 control-plane host，但不是 99 VMware 預設 VMX source。若 verifier stdout 已列出 `VMX alias=...` / `VM_POWER alias=...`，scorecard 必須以 verifier 實際 alias 集合為準。`windows99_vmx_source_repair_package.vm_rows` 必須是 default required aliases 與 runtime missing / powered-off aliases 的 union，避免 runtime 明確送進來的 alias 被 UI/API row 漏掉。此規則只修 source-of-truth / readback；不授權改 Windows task、relink、restore、啟動 VM 或重啟 host。 v1.119 Windows99 no-secret verifier via-host / validator rule：本工作站直連 99 SSH 缺 public-key auth 時，`collect-windows99-vmware-verify.sh` 必須支援 `--via-host wooo@192.168.0.110`，沿 110→99 已驗證 no-secret path 執行 Verify；collector 必須輸出 `via_host=`，仍固定 BatchMode/publickey、in-memory stdin scriptblock、`remote_write_performed=false`、`vm_power_change_performed=false`。`validate-windows99-console-verify-artifact.py` default required aliases 必須與 v1.118 一致為 `111 / 188 / 120 / 121 / 112`。2026-07-03 16:33 已證明舊 `110` default 會遮蔽 111 unreachable；不得再把舊 validator default 造成的 111 漏檢當真 green。 v1.120 Windows99 runtime metric overlay precedence rule：v1.110 的 nested blocker promotion 只適用於 runtime metric 缺少 Windows99 config / policy 欄位，或 metric 明確回 not ready 的情況。`reboot-auto-recovery-slo-exporter.sh` 必須從同一份 latest `scorecard.json` 輸出 `awoooi_windows99_vmware_autostart_config_ready`、`awoooi_windows99_vmware_windows_update_no_auto_reboot_ready`、`awoooi_windows99_vmware_missing_vmx_alias` 與 `awoooi_windows99_vmware_powered_off_alias`。production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 讀到 runtime metric `config_ready=1`、`windows_update_no_auto_reboot_ready=1`、`missing_vmx_aliases=[]` 時，不得再用 committed snapshot 的舊 `service_blockers` / `policy_blockers` 反加 `windows99_vmware_autostart_config_not_ready` 或 `windows99_update_no_auto_reboot_policy_not_ready`。若 `runtime_metric_active_blocker_count=7` 但 API `active_blocker_count` 被 stale snapshot 放大，優先修 API overlay / exporter metric，不得把假 blocker 轉成 Windows service restart、registry apply、VM power change、host reboot、Docker / Nginx / K3s / DB / firewall restart 或 secret 讀取。 v1.121 deployed Windows99 runtime metric overlay readback rule：2026-07-03 14:26 production readback 已證明 source `33c391808` 的 API overlay 與 110 textfile metric 對齊。Gitea CD `#4589` 為 `Success`，deploy marker `9f49651`，production deploy readback matched；110 最新 SLO run `20260703-142122` 回 `Result=success`，textfile metric 為 `awoooi_reboot_auto_recovery_slo_blocker_count=7`、`awoooi_windows99_vmware_autostart_config_ready=1`、`awoooi_windows99_vmware_windows_update_no_auto_reboot_ready=1`、missing VMX alias 空、powered-off aliases `110 / 112 / 120 / 121 / 188`。production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 必須讀回 `active_blocker_count=7`、`runtime_metric_active_blocker_count=7`、`runtime_metric_runtime_readback_added_blocker_count=0`、`readiness_percent=67`、`primary_blocker=reboot_event_required_host_unreachable`；若再次出現 Windows99 config / Windows Update policy blocker，視為 API / Prometheus / textfile regression 先修 readback，不得跳去 power on VM、改 Windows policy、重啟 host 或重啟服務。這不代表 10 分鐘自動恢復 SLO 完成；剩餘真 blocker 仍是 fresh all-host reboot event / 10-minute window / host reachability / Windows99 guest power readback。 v1.122 public maintenance edge fallback drift metric rule：deploy / reboot window 期間 AWOOOI API / SLO route 只要曾回 raw `502` 且缺 `X-AWOOOI-Fallback` / `Retry-After` / maintenance body，即使稍後自癒為 `200`，仍要保留 P0 blocker。`reboot-auto-recovery-slo-exporter.sh` 必須執行 `public-maintenance-edge-fallback-apply.sh --check`，把 live 188 edge Nginx drift merge 進 `public-maintenance-fallback.json`，並輸出 `awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"}`。當 edge check 顯示 `EDGE_FALLBACK_READY=0` 時，scorecard / API 必須新增 `edge_public_maintenance_live_config_drift`，讓 `public_maintenance_fallback.ready=false`；Alertmanager 必須透過 `PublicMaintenanceEdgeFallbackConfigDrift` 告警。此規則只授權 no-write check / metric / alert；實際 `--apply` 必須有 passwordless sudo、backup、`nginx -t`、reload 與 public header verifier，不得讀密碼、不得重啟 app / Docker / host。 v1.123 SLO installer edge-source packaging rule：110 SLO installer payload 必須包含 `ops/nginx`、`ops/maintenance` 與 `ops/monitoring`。若 110 runtime `public-maintenance-edge-fallback-apply.sh --check` 先回 `local_nginx_conf_missing`，那是 verifier source package 不完整，必須修 installer payload 並重新套用 verifier；不得把它當 live edge 已驗證或直接跳到 Nginx apply。2026-07-03 15:13 controlled install 已證明修正後 artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-151219` 正確讀到 188 live edge：`REMOTE_CONF_READABLE=1`、VIP `192.168.0.125` API/Web upstream 存在，但 `proxy_intercept_errors` / `error_page` / fallback header / internal maintenance location / maintenance file 均缺，`SUDO_N_READY=0`，textfile 輸出 `awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"} 0` 與 `edge_public_maintenance_live_config_drift`，blocker count `8`。這仍只授權 no-write verifier 與告警，不授權讀密碼、reload Nginx、重啟服務或 host。 v1.124 public maintenance route / edge blocker split rule：production API overlay 不得把 route fallback ready 與 edge Nginx fallback ready 視為同一件事。`public_maintenance_fallback.runtime_readback_present=true` 只能清除 `public_maintenance_fallback_runtime_readback_missing`；route 0 raw 5xx / 0 unreachable 只能清除 route-level blockers；只有 `edge_fallback_runtime_readback_present=true` 且 `edge_fallback_ready=true` 才能清除 `edge_public_maintenance_live_config_drift`。若 Prometheus textfile active label 已有 edge drift，但 public route probe 是 200 / ready 且缺 edge readback，`/api/v1/agents/reboot-auto-recovery-slo-scorecard` 必須保留 edge drift；否則會把下次 reboot / deploy raw 502 風險誤報成正常。此規則只修 scorecard/API readback，不授權 Nginx reload、app restart、Docker / host restart 或讀密碼。 v1.125 Windows99 autostart task run-state rule：`windows99-vmware-autostart.ps1 -Mode Verify` 必須在既有 `VMWARE_AUTOSTART_TASK ... ok=...` 之外輸出 `VMWARE_AUTOSTART_TASK_DETAIL` 與 `VMWARE_AUTOSTART_TASK_ACTION`，至少包含 principal、run level、logon type、target script present、last run time、last task result、next run time、action execute / argument。`last_task_result=267011` / `0x41303` 或 `last_run_time=1999-11-30T00:00:00` 代表 task exists/enabled 但尚未跑過，若同時 `powered_off_aliases` 非空，scorecard 必須新增 active blocker `windows99_vmware_autostart_task_never_ran`，textfile exporter 必須輸出 `awoooi_windows99_vmware_autostart_task_*` metrics，API overlay / action matrix 必須把 next action 指到 `run_windows99_vmware_autostart_check_mode_package_for_powered_off_aliases_then_rerun_no_secret_collector_no_vm_power_change`。2026-07-03 live no-secret via-110 readback 證明 `AWOOOI-Start-VMware-VMs` present/enabled/Ready，但 `last_task_result=267011`、`last_run_time=1999-11-30T00:00:00`；P0-4 真 target set 必須是 `111 / 188 / 120 / 121 / 112`，部署後重新收集 powered-off aliases，不得再沿用舊 `110` default。此規則只授權 readback、metrics、alert routing 與 check-mode package；不得直接啟動 VM、重啟 Windows host / VMware service、改 scheduled task、改 registry 或讀密碼 / secret。 v1.126 Windows99 bounded verifier / fallback compatibility rule：`windows99-vmware-autostart.ps1 -Mode Verify` 預設不得遞迴掃描 VM roots；111 VMX 缺失時要快速輸出 `VMX alias=111 present=0` 與 `VM_POWER alias=111 vmx_present=0 running=0`，深度尋找 VMX 必須走獨立 locator / check-mode lane。`reboot-auto-recovery-slo-exporter.sh` 的 Windows99 fallback artifact 必須與目前 default alias set `111 / 188 / 120 / 121 / 112` 完全相容；若舊 artifact 含 `110` 或缺 `111`，不得回灌 textfile / scorecard。這是避免 timeout 後用舊 110 artifact 假綠的防線；不授權 power change、relink、restore、host reboot 或 secret 讀取。 v1.127 Windows99 VMX repair CLI cold-start rule：`scripts/reboot-recovery/build-windows99-vmx-source-repair-package.py` 是 no-secret / no-DB / check-mode recovery tool，不得因工作站沒有 runtime `DATABASE_URL` 或 `.env` 就失敗；CLI 必須可用 harmless placeholder DB URL 完成 import 與 package build，但不得連 DB、不得讀 secret、不得寫 Windows、不得 power change。2026-07-03 17:18 production scorecard package readback：`work_order_id=windows99-vmx-source-repair-111`、`missing_vmx_aliases=["111"]`、`apply_allowed_by_this_package=false`、`rollback_required=true`、`vm_power_change_allowed=false`。同輪 via-110 locator 若回 `blocked_ssh_publickey_auth_missing` / `ssh_auth_probe_exit_status=124`，只能當 management-channel blocker；不得把 repair package ready 誤報成 111 VMX 已恢復。 v1.128 Windows99 no-secret verifier timeout budget rule：110→99 SSH channel ready 不等於 full VMware Verify 會在 90 秒內穩定完成。2026-07-03 17:34 live 對照：locator 經 via-110 成功收集，`expected_vmx_path_present_count=0`、`candidate_source_count=0`；同一條 verify collector 90 秒曾 timeout，但 180 秒成功回 `MISSING_VMX_ALIASES=111`、`VMWARE_AUTOSTART_CONFIG_READY=0`、`VMWARE_AUTOSTART_VERIFY_READY=0`。因此 `collect-windows99-vmware-verify.sh` 與 SLO exporter 的 Windows99 remote verify 預設 timeout 必須是 `180` 秒；timeout 超過仍 fail-closed，但不得用 90 秒假 timeout 改變 P0 方向。此規則不授權 VM power change、Windows write、host reboot、service restart 或 secret 讀取。 2026-07-02 110 control-path / Harbor recovery receipt rule：若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure，但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`，則該 Gitea Harbor repair 失敗只能列為 historical queue metadata，不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier，並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示：public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`；active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure，以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker，non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker；non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。 2026-07-02 110 controlled CD lane fail-closed enforcer staging rule：110 runner 壓力事故後，legacy / generic runner 仍必須 fail-closed；但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail，且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時，才可保留 drain config / binary / unit，並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service；若 registration 缺失，readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走，優先修 source enforcer / unit guardrail，不要手工反覆補同一組 artifact。 2026-07-01 23:00 latest live summary：core cold-start 已從 degraded 收斂為 GREEN，但仍不可宣稱 DR complete 或 MOMO 業績資料已最新。`full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` artifact `/tmp/awoooi-cold-start-source-gate-20260701-225720.log` 回 `PASS=96 WARN=0 BLOCKED=0`、`Result: GREEN`。MOMO daily stale 在 `MOMO_DAILY_FRESHNESS 7|2026-06-24` 且 no-newer-source evidence 成立時不再算 core cold-start warning：cold-start 會輸出 `OK 188 momo daily sales source gate has no newer Drive candidate` 與 `INFO 188 momo daily sales data remains stale; product data freshness is pending source arrival`；這表示主機/服務恢復完成，但產品資料 freshness 仍留在 source-arrival gate，必須等正式 Drive source 到達後由原匯入 pipeline 更新，不得手動 DB 偽更新。110 live monitor 已同步，`/home/wooo/scripts/full-stack-cold-start-check.sh` hash `6115f73002b7e5b0fc46a031a2e7e9049d68abfcc8110f638e975218792c468e`；110 textfile 讀回 `awoooi_cold_start_monitor_up=1`、`pass=96`、`warn=0`、`blocked=0`、`last_exit_code=0`、`last_result{result="green"}=1`、`last_run_duration_seconds=26`。`verify-cold-start-monitor-deploy.sh` 回 `COLD_START_MONITOR_DEPLOY_PARITY_OK`，runtime state `monitor_up=1 warn=0 blocked=0 green=1 blocked_state=0`，ColdStart alerts `0`。`full-stack-recovery-scorecard.sh` 回 `CORE_COLD_START_GREEN=1`、`CORE_COLD_START_WARN_GATES=0`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_FIRING_ALERTS=0`、`CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`ESCROW_MISSING_COUNT=5`、`NEXT_STEP=complete_credential_escrow_review`、`RECOVERY_STATE=CORE_READY_DR_OFFSITE_PENDING`。Allowed declaration：110 / 120 / 121 / 188 core cold-start service recovery GREEN，public routes / AWOOOI service / Gitea / Harbor registry / K3s / Stock public route / 188 backup-from-110 / 110 awoooi_db freshness 已恢復。Forbidden declaration：DR complete、credential escrow complete、MOMO data 最新、110 SSH 永久穩定、以假資料或手動 DB 寫入掩蓋 source freshness。 2026-07-01 21:32 previous live summary：cold-start 假 WARN 已收斂，hard blockers 維持 `0`，但仍不可宣稱 full green 或 10 分鐘全自動恢復完成。`full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` final artifact `/tmp/awoooi-cold-start-final-20260701-212632.log` 回 `PASS=95 WARN=1 BLOCKED=0`；public routes / TLS 全部通過，StockPlatform 21:22 左右的 `502` 已確認是 web/admin/edge 替換 warmup，外部連續 5 次 `https://stock.wooo.work/` 回 `200`，final cold-start 亦回 `stock 200`。K3s `BAD_PODS=2` 也是 rollout 暫態，連續 6 次只讀觀察已無非 Running/Completed pod，final `BAD_PODS 0`。MOMO current-month `0|0|-|-|-|-` 不再列為 WARN：`momo-drive-token-source-recovery-preflight.sh` 會輸出 `MOMO_LATEST_IMPORT_CLEAN` 與 `MOMO_SOURCE_ABSENT_WITHOUT_NEWER_DRIVE`，cold-start 讀到 latest clean import 且 Drive 無更新 source candidate 時，判定 current-month sync not applicable。110 backup current health 也不再被舊 aggregate log 壓成 WARN：`BACKUP_HEALTH_110 total=13 stale=0 missing_cron=0 missing_script=0 failed_count=5 config_failed=0 integrity_total=2 integrity_stale=0` 代表 current component freshness / critical config / integrity OK；`failed_count=5` 保留為 INFO evidence，等下一次 full `backup-all` 自然覆蓋。live 110 monitor 已同步，hash `full-stack-cold-start-check.sh=d0711f75dfb1ee680442c9d6cf2191741f3b27605f347c9ef2a25a4fed6d40ac`、`momo-drive-token-source-recovery-preflight.sh=571d75e81c509683eb8a38fabbe81fc7822befe45206145f4fb4e865473f5254`；110 textfile 讀回 `awoooi_cold_start_monitor_up=1`、`pass=95`、`warn=1`、`blocked=0`、`last_exit_code=1`、`last_result{result="degraded"}=1`。`verify-cold-start-monitor-deploy.sh` 回 `COLD_START_MONITOR_DEPLOY_PARITY_OK`；`full-stack-recovery-scorecard.sh` 回 `CORE_COLD_START_WARN_GATES=1`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`ESCROW_MISSING_COUNT=5`、`NEXT_STEP=complete_credential_escrow_review`。Allowed declaration：public routes / AWOOOI service / Gitea / Harbor registry / 188 backup-from-110 / 110 awoooi_db freshness / K3s rollout / Stock public route 已恢復，cold-start hard blockers `0`。Forbidden declaration：full green、10 分鐘全自動恢復完成、DR complete、credential escrow complete、MOMO data 最新、110 SSH 永久穩定。唯一 cold-start WARN 是 MOMO daily data freshness：`MOMO_DAILY_FRESHNESS 7|2026-06-24`，且 Drive intake / failed folder 無新候選；必須走 source-arrival / formal import gate，不可用假資料或手動 DB 寫入掩蓋。 2026-07-01 21:05 previous live summary：backup / DR freshness 已收斂，但仍不可宣稱 full green 或 10 分鐘全自動恢復完成。188 `backup_from_110` 已受控重跑成功，Harbor / Gitea / bitan backup 均 OK；188 exporter 讀回 `backup_from_110 fresh=1`、Gitea private bundle `expected_repo_missing_count=0`、`failed_repo_count=0`。110 `awoooi_db` high-frequency backup 20:00 因 `too many connections for role "awoooi"` 失敗；本輪在 DB 舊 OR 查詢為 `0`、Postgres CPU 已低的條件下走 controlled short window，產生 Restic snapshot `211a8948`，時間 `2026-07-01T20:58:20+08:00`，tags `service:awoooi,freq:6h,timestamp:20260701_204344`，tmp dump 已清除。正式保守調整為 `awoooi CONNECTION LIMIT 4`，保留 `statement_timeout=750ms`、`max_parallel_workers_per_gather=0`、`enable_seqscan=off`；rollback 是 `ALTER ROLE awoooi CONNECTION LIMIT 2` 並保留同樣 role config。110 backup exporter 讀回 `awoooi_backup_job_fresh{job="awoooi_db"} 1`、`awoooi_backup_dr_phase{next_step="complete_credential_escrow_review"} 3`、`awoooi_backup_dr_credential_escrow_missing_count 5`。`full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` 回 `PASS=93 WARN=3 BLOCKED=0`；`full-stack-recovery-scorecard.sh` 回 `CORE_COLD_START_WARN_GATES=3`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`NEXT_STEP=complete_credential_escrow_review`。Allowed declaration：public routes / AWOOOI service / Gitea / Harbor registry / 188 backup-from-110 / 110 awoooi_db freshness 已恢復，cold-start hard blockers `0`。Forbidden declaration：full green、10 分鐘全自動恢復完成、DR complete、credential escrow complete、MOMO data 最新、110 SSH 永久穩定。剩餘 WARN 是 MOMO current-month/source freshness warning 與 110 aggregate `backup-all` 上次 failed components。高頻全量 `pg_dump` 本輪耗時 `1152s` 並產生 `28G` dump，後續 P0 必須建立專用 backup DB role / `AWOOOI_BACKUP_DATABASE_URL`，並把大型可重建 snapshot 表改成分層備份或 retention 後納入 full backup；不得把 app role 放寬當長期唯一解法。 2026-07-01 19:32 latest live summary：110 卡住的根因已確認並修復。主機 ping / SSH / Prometheus / Gitea / registry 均可達，但 `awoooi-startup-110.service` 因 `/usr/local/bin/awoooi-startup-110.sh` 被替換成 runner/CD lane fail-closed stub 而固定 `exit 75`，造成 post-reboot systemd degraded 與 cold-start warning。live stub hash 為 `55e1e87d44cf5b8cefb714a49d085b886697d856569ce9478537059d49129f88`，repo 完整 startup script hash 為 `fcc67c7dde889b3cf8ffeca89cc5e58407fabc6f5a80554e39f3f1465f90b318`。修復方式固定使用 `scripts/reboot-recovery/repair-110-startup-script-stub.sh`：預設 `--verify` 只讀；`--apply` 只上傳 repo 完整 startup script、備份 live stub、解除 immutable、install 到 `/usr/local/bin/awoooi-startup-110.sh`、`systemctl daemon-reload`，並只 `reset-failed awoooi-startup-110.service`；`--start-service` 需另外明確指定，因為它會跑完整 startup checks。live apply 指令為 `BACKUP_SUFFIX=before-full-startup-restore-20260701-1928 bash scripts/reboot-recovery/repair-110-startup-script-stub.sh --apply`，post-readback `remote_matches_expected=1`、`remote_is_runner_failclosed_stub=0`，rollback backup 為 `/usr/local/bin/awoooi-startup-110.sh.before-full-startup-restore-20260701-1928`。之後清除 stale `systemd-ask-password-console.path` / `systemd-ask-password-wall.path` failed state；110 已回 `systemctl is-system-running=running`、failed units `0`。最新 cold-start 讀回 `PASS=90 WARN=6 BLOCKED=0`、`FAILED_UNITS_110 0`；110 startup stub / systemd degraded 不再是 blocker。禁止再把 live `/usr/local/bin/awoooi-startup-110.sh` 當 runner/CD lane opener 封成 stub；runner fail-closed enforcer 只能封 runner binary、runner unit、controlled opener artifact 或 upload opener source，不得封 host startup recovery script。Allowed declaration：110 不再卡在 startup stub / systemd degraded，cold-start hard blockers `0`。Forbidden declaration：full green、10 分鐘全自動恢復已完成、DR complete、credential escrow complete、MOMO data 最新或 110 SSH 永久穩定；剩餘 warnings 仍需由 backup / DR escrow / alert 收斂處理。 2026-07-01 19:13 live summary：全主機重啟後的 service hard blockers 已清零，但仍不可宣稱 DR complete 或 full green。`full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1` 回 `PASS=89 WARN=7 BLOCKED=0`；public routes / TLS 含 `https://signoz.wooo.work/` 均回 expected 2xx/3xx，AWOOOI API health `200`，Gitea version API `200`，Gitea CD latest visible `#4283` 為 `Success`。MOMO dedicated source preflight 已從 hard blocker 降為 source freshness warning：`MOMO_DRIVE_TOKEN_SOURCE_PREFLIGHT PASS=19 WARN=6 BLOCKED=0`、`DB_DAILY_FRESHNESS 7|2026-06-24`、`DRIVE_INTAKE_COUNT=0`、`DRIVE_FAILED_COUNT=0`、`DRIVE_GLOBAL_LATEST_MODIFIED=2026-06-25T04:21:47Z`、latest import job `57 completed` 且 `15383|15383|0`，因此「沒有比最後乾淨 import 更新的 Drive source」不得再硬擋 cold-start；若未來 auth failure、failed folder 有候選、新 source 未匯入或 latest import 不乾淨，仍 hard block。110 cold-start monitor deploy parity 已擴充到 `full-stack-cold-start-check.sh`、`cold-start-textfile-exporter.sh` 與 `momo-drive-token-source-recovery-preflight.sh` 三者 hash；latest remote textfile metric 為 `awoooi_cold_start_pass_gates=89`、`warn=7`、`blocked=0`、`last_result{result="degraded"}=1`。SignOz 的 root-owned Nginx source drift 已在 repo 修成 `192.168.0.110:8080`，但 188 `ollama` sudo 仍需要密碼；為了不讀密碼、不 reload Nginx，本輪用 user-level controlled bridge 先恢復 live route：`scripts/reboot-recovery/signoz-188-upstream-bridge.sh --apply` 在 188 建立 `/home/ollama/bin/awoooi-signoz-upstream-bridge.sh` 與 user crontab `@reboot`，以 `socat` 監聽 `127.0.0.1:3301` 並轉到 `192.168.0.110:8080`；status readback `SIGNOZ_BRIDGE_STATUS running=1 listen_code=200 upstream_code=200`，rollback 是 `scripts/reboot-recovery/signoz-188-upstream-bridge.sh --rollback`。19:13 scorecard 讀回 `CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`CORE_COLD_START_BLOCKED_GATES=0`、`CORE_COLD_START_WARN_GATES=7`、`CORE_COLD_START_FIRING_ALERTS=2`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`ESCROW_MISSING_COUNT=5`、`RECOVERY_STATE=CORE_NOT_READY_DR_OFFSITE_PENDING`。Allowed declaration：網站 / public routes / AWOOOI service / Gitea / Harbor registry path / SignOz public route 已恢復，cold-start hard blockers `0`。Forbidden declaration：10 分鐘全自動恢復已完成、full green、DR complete、credential escrow complete、MOMO business data 最新、110 SSH 永久穩定或 Nginx privileged source apply 已完成。下一步固定為 P0 backup/DR escrow/alert warning 收斂：credential escrow non-secret evidence 仍缺 `5`，backup health 仍有 stale / failed-component warning。 2026-07-01 18:40 previous live summary：全主機重啟後仍不可宣稱 10 分鐘自動恢復，但主 blocker 已從 110/Harbor/Gitea source parity 轉為實際 runtime/data blocker。repo-side readiness 已修成 `PASS=199 WARN=1 BLOCKED=0`；`full-stack-cold-start-check.sh --monitor-read-only --no-color` 回 `PASS=86 WARN=8 BLOCKED=2`。110 P0 區段已讀回 Harbor / Gitea / Prometheus / Alertmanager / Sentry OK、legacy/direct runner fail-closed、controlled CD lane fail-closed、root restore source left `0`、storage clean、textfile exporters fresh；但 `wooo@192.168.0.110` command channel 曾在同輪 backup 前 timeout，bounded retest 又回 `SSH_COMMAND_OK`，因此要記為 intermittent 110 control-channel evidence，不得宣稱 110 SSH 永久穩定。110 live cold-start monitor 已用既有 installer 同步，rollback evidence 保留在 `/home/wooo/scripts/full-stack-cold-start-check.sh.before-p0-readiness-20260701-183215` 與 `/home/wooo/scripts/full-stack-cold-start-check.sh.before-hostkey-policy-20260701-183637`；live hash 為 `full-stack-cold-start-check.sh=e320c061f5afd31c2a682576218f549b683f25dafd43dd52acc13b6283b33712`、`cold-start-textfile-exporter.sh=c52ea4fe8dd58688a87c01ca6288f8f6050aeb82417852213db3e2be69b29568`。`verify-cold-start-monitor-deploy.sh` 現在只判斷 deploy parity：hash、host-key policy、monitor-up；runtime green 另由 cold-start / scorecard 判斷，避免 SignOz/MOMO 尚未恢復時把 source parity 誤報成 deploy mismatch。18:40 scorecard 讀回 `CORE_COLD_START_DEPLOY_PARITY=1`、`CORE_REGISTRY_READY=1`、`CORE_COLD_START_BLOCKED_GATES=2`、`CORE_COLD_START_FIRING_ALERTS=3`、`DR_OFFSITE_EVIDENCE_READBACK=1`、`ESCROW_MISSING_COUNT=5`、`RECOVERY_STATE=CORE_NOT_READY_DR_OFFSITE_PENDING`。目前 hard blockers 是 `signoz.wooo.work` public 502 / TLS failure，以及 `188 momo daily sales data stale beyond 3 days`；DR 仍缺 5 個 credential escrow non-secret evidence。18:47 source fix 已把 SignOz source-of-truth 從 `127.0.0.1:3301` 改到實際 110 upstream `192.168.0.110:8080`，且 cold-start probe 改為檢查同一 upstream；live apply 仍未執行，因為 188 `sudo -n nginx -t` 回 `sudo: a password is required`。不可宣稱：full-stack green、10 分鐘全服務恢復、MOMO daily data 最新、SignOz public route 正常、DR complete 或 110 SSH 永久穩定。下一步固定為 privileged Nginx config apply/readback 或本機 console 套用 SignOz route，接著處理 MOMO source freshness；110 若再次 command timeout，走 local console / control-channel recovery package，不重啟主機、不恢復 generic runner。 2026-06-30 22:55 latest live summary：全主機重啟後仍不可宣稱 10 分鐘自動恢復。`SSH_COMMAND_TIMEOUT_SECONDS=8 SSH_BATCH_MODE=yes bash scripts/reboot-recovery/full-stack-cold-start-check.sh --monitor-read-only --no-color` artifact `/tmp/awoooi-cold-start-live-after-ff.log` 回 `PASS=68 WARN=4 BLOCKED=4`，hard blockers 是 110 registry external `/v2`、110 SSH read-only check、K3s registry pull refused by `110:5000`、SignOz TLS / 502。StockPlatform public freshness / ingestion 22:50 仍回 `status=not_configured`、`blockers=["postgres_not_ready"]`。Public Gitea queue 22:55 回 `status=blocked_harbor_110_repair_no_matching_runner`，latest CD `#4105` 雖顯示 `Running`，但 build log 已有 `latest_visible_cd_inflight_classifier=harbor_registry_public_route_unavailable_pending_retry`、`latest_visible_cd_harbor_latest_registry_v2_status=502`、`latest_visible_cd_harbor_public_route_retrying_unavailable=true`、`harbor_controlled_repair_skipped=not_110_host`；Harbor repair 仍 `Waiting` 且缺 `awoooi-host` runner。判定：CD `Running` 不得視為中性等待；若 registry retry 已連續 502 / 000 且 repair 無 110 control path，立即依 110 control path / Harbor `/v2` 主線 blocker 處理。不可宣稱：全服務恢復、最新版本已上 production、Stock 資料最新、backup core green、DR complete、188 hygiene green。下一步仍固定為恢復 110-local repair control path / Harbor `/v2`，再重跑 post-reboot summary、cold-start、Stock freshness / ingestion 與 SLO scorecard。 v1.83 Gitea CD running retry rule：`read-public-gitea-actions-queue.py --json` 必須同時看 `latest_visible_cd_failure_classifier` 與 `latest_visible_cd_inflight_classifier`。final `BLOCKER harbor_registry_public_route_unavailable` 尚未出現時，只要 `latest_visible_cd_harbor_public_route_retrying_unavailable=true` 且 `latest_visible_cd_harbor_latest_registry_v2_status` 不是 `200/401`，就把它當作 in-flight production deploy blocker evidence；若 Harbor repair workflow 同時 `Waiting` 或 no-matching `awoooi-host`，下一步是恢復 110 local repair control path，而不是等 CD timeout、重跑無效 CD、workflow_dispatch，或把 `Running` 當作版本最新。 2026-07-01 07:58 live host-pressure update：188 持續高 CPU 的原因不是一般重啟噪音，而是 `k3s-postgres-recovery` 內 `awooop_conversation_event` hot-path index drift。live DB 原本只剩 `awooop_conversation_event_pkey`、`idx_conv_event_run`、`idx_conv_event_subject`、`uix_conv_event_dedup`，缺 base `idx_conv_event_recent` 與 provider/source_refs hot-path indexes；`k3s-postgres-recovery` 當時約 `7.9277` CPU cores、188 `load5=10.57`。已依 `apps/api/migrations/awooop_conversation_event_hot_path_indexes_2026-07-01.sql` 走 controlled DB migration，僅執行 `CREATE INDEX CONCURRENTLY IF NOT EXISTS`，`lock_timeout=5s`，無 DROP / TRUNCATE / restore / DB restart / Docker restart / secret read。post-apply verifier 顯示 12 個新索引全部 `indisvalid=true`、`indisready=true`；`pg_stat_activity` 收斂到 active `1`，第一次讀回 `k3s-postgres-recovery` 降到約 `1.0552` cores，20 秒後已不在 188 top 3 CPU containers。receipt：`docs/operations/awooop-conversation-event-hot-path-index-apply-receipt-2026-07-01.snapshot.json`。110 仍高不是同一個 DB 問題：110 `gitea` 仍約 `3.4019` cores，public queue 仍是 `blocked_harbor_110_repair_no_matching_runner` / `awoooi-host`；下一步固定為 110 Gitea queue / controlled lane recovery，不得恢復 generic runner、不得重啟主機。 v1.82 bounded summary rule：`post-start-quick-check.sh` 與 `188-host-hygiene-maintenance-checklist.sh` 的 SSH helper 必須有 command timeout、single connection attempt、ServerAlive 與 no password prompt；任何 110 / 188 read-only control path 卡住時，都要收斂成 blocker / evidence，而不是讓 `post-reboot-readiness-summary.sh` 無限等待。若 backup / escrow 證據讀不到，`ESCROW_MISSING_COUNT=unknown` 必須同時輸出 `DR_ESCROW_BLOCKED=1` 與 `DR_ESCROW_EVIDENCE_UNKNOWN=1`，並把 `backup_core_readback_recovery`、`credential_escrow_evidence` 放進 `NEXT_REQUIRED_GATES`；unknown 不得被解讀為 DR 或 backup green。 2026-06-29 09:13 previous live summary：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` artifact `/tmp/awoooi-post-reboot-readiness-20260629-091918/summary.txt` 回傳 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_SERVICE_WARNINGS=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`STOCK_FRESHNESS_STATUS=ok`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`BACKUP_CORE_GREEN=1`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=6`、`RUNTIME_ACTION_AUTHORIZED=0`、`NEXT_REQUIRED_GATES=credential_escrow_evidence`。此 baseline 已被 2026-06-30 20:18 全主機重啟後 evidence 覆蓋，不得再拿來宣稱目前 green。 2026-06-27 11:51 最新 live revalidation：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` artifact `/tmp/awoooi-post-reboot-readiness-20260627-115046/summary.txt` 回傳 `POST_START_RESULT=BLOCKED`、`POST_START_PASS=37`、`POST_START_WARN=3`、`POST_START_BLOCKED=2`、`SERVICE_GREEN=0`、`PRODUCT_DATA_GREEN=1`、`STOCK_FRESHNESS_STATUS=ok`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`STOCK_BLOCKERS=none`、`BACKUP_CORE_GREEN=1`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`RUNTIME_ACTION_AUTHORIZED=0`。本輪已再次修復 188 `momo_pg_daily` crontab configured drift，`backup-status` 回 `core_blockers=0`、`configured_missing_188=0`；K3s / ArgoCD live readback 顯示 120 / 121 皆 `Ready`，`awoooi-prod` 為 `Synced / Healthy`，api/web/worker pods 均 Running。現在 hard blocker 是 MOMO business data freshness：`daily_sales_snapshot` 最新仍為 `2026-06-24`，`DRIVE_INTAKE_COUNT=0`，Drive archive / global latest `即時業績_當日` 均為 `2026-06-25T04:21:47Z`，最新 import job `57` 已 clean completed 且 `sync_success=true`。因此可宣稱主機、K3s、public routes、backup core 與 Stock freshness 已恢復；不可宣稱 full-stack green，直到 MOMO 來源檔補齊並由正式 import pipeline 更新 DB。DR complete 仍因 `ESCROW_MISSING_COUNT=5` 禁止宣稱，Wazuh 全主機納管仍因 manager registry accepted `0` 禁止宣稱。 2026-06-27 00:58 最新 live summary：本輪先修復兩個實際 SOP blocker。第一，`scripts/ops/recovery-scorecard-contract-check.py` 已改成 PyYAML optional，標準 Python 環境也能驗證 recovery recording-rule contract，不會因 `ModuleNotFoundError: yaml` 中斷 DR/offsite checklist。第二，188 `ollama` crontab 已備份到 `/home/ollama/momo_backups/crontab-before-momo-pg-host-owned-20260627-001925.txt`，並把 `AWOOOI momo PostgreSQL daily backup` 從 app-side `/home/ollama/momo-pro/scripts/pg_backup.sh` 收斂回 host-owned `/home/ollama/bin/momo-pg-backup.sh`；刷新 188 textfile exporter 後 `awoooi_backup_job_configured{host="188",job="momo_pg_daily"} 1`。00:58 `scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` artifact `/tmp/awoooi-post-reboot-readiness-20260627-005728/summary.txt` 回傳 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_PASS=38`、`POST_START_WARN=3`、`POST_START_BLOCKED=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`RUNTIME_ACTION_AUTHORIZED=0`。同輪 `backup-status` 回 `core_blockers=0`、`configured_missing_188=0`；Prometheus live contract 回 `awoooi_recovery_core_ready=1`、`awoooi_recovery_dr_offsite_ready=0`，表示主機 / K3s / public routes / product data / backup core 已恢復，DR 仍只因 credential escrow 缺 5 個 non-secret evidence marker blocked，Wazuh 全主機 registry accepted 仍為 0。 2026-06-27 00:02 最新 live summary：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 回傳 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_PASS=38`、`POST_START_WARN=4`、`POST_START_BLOCKED=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`STOCK_FRESHNESS_STATUS=ok`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`STOCK_BLOCKERS=none`、`BACKUP_CORE_GREEN=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`RUNTIME_ACTION_AUTHORIZED=0`。同一輪 production route smoke 回傳：IwoooS `200`、Wazuh read-only routes `200`、VibeWork `200`、AwoooGo `200`、MOMO health `200`、Stock `200`；AWOOOI API health `healthy / prod / mock_mode=false`，PostgreSQL / Redis / OpenClaw / SigNoz / GCP Ollama provider up，local Ollama endpoint 仍為 cooldown / degraded，由 provider fallback 承接，不是網站或 API service blocker。最新 deploy marker 為 `e506b9d5 chore(cd): deploy fe74d86 [skip ci]`；本輪 `89b9e67a` 是 SOP / scripts / docs source update，不是 runtime bundle deploy marker。112 Wazuh 與 120 K3s 的 23:56 脫敏 readback 仍作為本輪相鄰 evidence：120 ArgoCD `Synced / Healthy`、Pod 均 `Running` 或 `Completed`；Wazuh manager registry 並非全空，但 `WAZUH_MANAGER_REGISTRY_ACCEPTED=0` 維持，不能宣稱全主機納管恢復。 2026-06-26 23:56 live summary retained for comparison：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 回傳 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_PASS=38`、`POST_START_WARN=3`、`POST_START_BLOCKED=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`STOCK_FRESHNESS_STATUS=ok`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`STOCK_BLOCKERS=none`、`BACKUP_CORE_GREEN=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`RUNTIME_ACTION_AUTHORIZED=0`。同一時段只讀補查 120：ArgoCD `awoooi-prod` 為 `Synced / Healthy`，`awoooi-prod` Pod 均為 `Running` 或 `Completed`；歷史 `km-vectorize-29689620` failed Job 已被 2026-06-23、2026-06-24、2026-06-25 後續成功 Job 覆蓋，不是目前服務 blocker。同一時段只讀補查 112：systemd `running`，Wazuh manager / indexer / dashboard `active`，manager API root 回 `401`，Dashboard unauthenticated check endpoints 回 `401`，manager registry 脫敏讀回為 local manager `1`、受管 agent `5`、active managed `5`、disconnected `0`、never connected `0`。此證據證明 registry 不再是「全空」，但仍不能宣稱 Wazuh 全主機納管恢復，因為 SOP expected scope 仍是 6、Dashboard API connection / version 尚未以登入或 owner evidence 驗收，owner response accepted 仍為 `0`。 2026-06-26 18:46 最新即時恢復真相已覆蓋 12:13 對今日產品資料的判讀：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 回傳 `POST_START_RESULT=PRODUCT_DATA_PENDING_EOD_WINDOW`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=0`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`STOCK_BLOCKERS=core_margin_short_daily_missing,ai_recommendations_stale`、`BACKUP_CORE_GREEN=1`、`ESCROW_MISSING_COUNT=5`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`。同一輪 live cold-start 長檢查回傳 `PASS=87 WARN=0 BLOCKED=0` 與 `Result: GREEN`，代表 110 / 120 / 121 / 188 主機、K3s、public routes、AWOOI API、MOMO、backup core、exporters、cron 與 Alertmanager 服務層已恢復；但 StockPlatform 今日官方 margin-short 尚未發布，AI recommendations 仍依賴該資料，因此不可宣稱所有產品資料最新。18:43 已以授權 `SIGTERM` 清除 110 上兩組 6 小時以上 `stockplatform-review-bulk-ux` orphan Chrome process group，`REMAINING=0`；18:44-18:46 已停止 168 Mac Mini 上本機 AWOOOI `next build` 並清理 temp/build/cache 與 Antigravity backup browser recordings，使 `/System/Volumes/Data` 從約 `1.0Gi / 100%` 回到約 `8.7Gi / 96%`。112 Kali 的 `networking.service` failed 已定位為 `/etc/network/if-up.d/wg-nat` 錯誤 shebang `#\!/bin/bash` 導致 `Exec format error`；Wazuh manager / indexer / dashboard 仍 active，該 hook 修復需要 112 sudo 提權，未使用或保存密碼。 2026-06-26 12:13 latest live summary supersedes the 08:59 gate set：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 回傳 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_PASS=38`、`POST_START_WARN=4`、`POST_START_BLOCKED=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`DR_ESCROW_BLOCKED=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_SERVICE_GREEN=1`、`HOST_188_HYGIENE_BLOCKED=0`、`HOST_188_RESULT=HOST_188_HYGIENE_GREEN.`、`WAZUH_ROUTE_CODE=200`、`WAZUH_TRANSPORT_COUNT=6`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`WAZUH_DASHBOARD_API_CONNECTION=pending_or_spinning`、`WAZUH_DASHBOARD_INDEX_OK=3`、`RUNTIME_ACTION_AUTHORIZED=0`、`OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`NEXT_REQUIRED_GATES=credential_escrow_evidence,wazuh_manager_registry_export`。188 host hygiene 已從 blocker 移除；目前不可宣稱完成的只剩 DR credential escrow 與 Wazuh manager registry。ACME HTTP-01 route 與 certbot timer hygiene 已修復，但不得宣稱憑證已正式 renew，需等 snap certbot timer / ACME window readback。 2026-06-26 13:01 owner response preflight baseline：新增 `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color` 與 `docs/templates/post-reboot-next-gate-owner-response.json`。無 response file 時必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_file expected_gates= received=0 accepted=0 runtime_gate=0`；直接使用 placeholder template 時必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_content expected_gates= received=0 accepted=0 runtime_gate=0`。2026-06-26 當時 active gate count 為 2；現行 gate count 以 v1.79 summary / owner packet 為準。此 gate 只驗收 active gates 的脫敏 owner evidence，不送 request、不寫 escrow marker、不讀 secret、不做 Wazuh / host / Kali runtime action，也不把一般批准訊息轉成 owner accepted。 2026-06-26 17:45 single-summary replay baseline：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 現在會自動寫入 `/tmp/awoooi-post-reboot-readiness-20260626-174451/summary.txt`，同一輪後續 `declaration guard`、`next-gate dispatch`、`owner packet`、`contract guard` 與 `owner response preflight` 均用此 summary 重放。17:45 summary 回傳 `SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`DR_ESCROW_BLOCKED=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`NEXT_REQUIRED_GATES=credential_escrow_evidence,wazuh_manager_registry_export`。`post-start-quick-check.sh` 也已補 route warmup 分類：若 delegated cold-start 的 `BLOCKED` 全部是 public route，且 wrapper 自己的 route retry 已全部恢復，該 cold-start blocker 會降級為 evidence warning，不再把整輪服務恢復誤判成 blocked；非 route blocker 或 retry 後仍失敗仍維持 hard blocked。 2026-06-26 07:47 machine-readable readiness summary retained as historical pre-repair evidence：當時 `HOST_188_HYGIENE_BLOCKED=1`、`NEXT_REQUIRED_GATES=credential_escrow_evidence,host_188_hygiene_maintenance_window,wazuh_manager_registry_export`。此段只用來比對 188 修復前後差異；現行 gate set 必須使用 12:13 baseline。 2026-06-26 08:12 next-gate dispatch baseline retained as historical pre-repair evidence：當時 output 固定三個 P0 checklist。12:13 起 dispatch 依 live summary 動態輸出，當日 example 為 `NEXT_GATE_COUNT=2`、credential escrow 與 Wazuh registry；現行 gate count 必須以同一輪 summary / owner packet 為準。 2026-06-26 08:29 owner-packet JSON baseline：`scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color` 將 dispatch output 轉成 `schema_version=awoooi_post_reboot_next_gate_owner_packets_v1`，包含三個 `owner_packets`、`next_gate_count=3`、`p0_gate_count=3`、`request_sent_count=0`、`owner_response_received_count=0`、`owner_response_accepted_count=0`、`runtime_action_authorized_count=0`。此 JSON 是 AI / operator / owner review intake，不是外部 request，也不是維護窗口批准。 2026-06-26 08:40 owner-packet contract guard baseline retained as historical pre-repair evidence：舊版鎖定三個 P0 gate。12:13 起 contract guard 依 `source.next_required_gates` 動態驗收；2026-06-26 artifact 當時是 `POST_REBOOT_OWNER_PACKET_CONTRACT_GUARD_OK gates=2 request_sent=0 accepted=0 runtime_gate=0`，現行 expected line 以同一輪 summary / owner packet 為準。若 188 hygiene future regression，才會重新增加對應 gate。 2026-06-26 08:47 Wazuh registry detail baseline：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 已把 Wazuh repo-side coverage / runtime gate 的細節納入固定 key/value：`WAZUH_COVERAGE_SCOPE=6`、`WAZUH_DIRECT_ACTIVE=2`、`WAZUH_NO_TRANSPORT=1`、`WAZUH_SSH_BLOCKED=3`、`WAZUH_ROUTE_CODE=200`、`WAZUH_TRANSPORT_COUNT=6`、`WAZUH_DASHBOARD_API_CONNECTION=pending_or_spinning`、`WAZUH_DASHBOARD_INDEX_OK=3`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`WAZUH_RUNTIME_GATE=0`。`scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color` 的 `wazuh_manager_registry_export` gate 會把這些狀態放入 `CURRENT_EVIDENCE`。判讀鐵律：route `200`、transport `6`、Dashboard index pattern `3` 都不是 manager registry accepted；全主機納管與 Dashboard API 修復仍需 owner evidence / registry export / acceptance record。 2026-06-26 08:59 declaration guard baseline retained as historical pre-repair evidence：當時 `HOST_188_FULLY_GREEN` 仍 forbidden。12:13 起 guard 依 `HOST_188_HYGIENE_BLOCKED=0` 動態允許 188 host hygiene green，但仍拒絕 `DR_COMPLETE`、`WAZUH_REGISTRY_RECOVERED`、`RUNTIME_ACTION_AUTHORIZED`。 2026-06-26 07:39 live quick-check refresh：`scripts/reboot-recovery/post-start-quick-check.sh --no-color` 完整跑完，四主機 ping / SSH 全部 OK，delegated cold-start 為 `PASS=89 WARN=0 BLOCKED=0`，wrapper 總結為 `POST_START_QUICK_CHECK PASS=38 WARN=3 BLOCKED=0`、warning split `SERVICE=0 BOUNDARY=1 EVIDENCE=2`、`RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。MOMO health `V10.701`，daily snapshot `109061` rows / `2025-07-01..2026-06-24`，current-month parity `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`，latest import job `57 completed`。StockPlatform freshness `status=ok`、latest trading date `2026-06-25`，price / chips / margin / AI recommendations 均為 `2026-06-25`。Backup-status 07:39 顯示 110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、offsite/rclone fresh、`last_backup_all=2026-06-26 02:31:02`、`escrow_missing=5`。Public routes extended list 全部回 expected 2xx/3xx。110 CPU attribution 顯示 load 約 `5.19 / 4.66 / 4.91`，CPU idle 多數樣本 `80%+`，目前負載來自 Gitea / ClickHouse / Docker / Kafka / StockPlatform / AWOOOI API / Sentry 等正常平台工作，不是 orphan Chrome。這一輪 allowed declaration：主機、K3s、服務、網站、產品資料 freshness、備份核心與 offsite freshness 綠；forbidden declaration：DR complete、credential escrow complete、188 host fully green、Wazuh registry recovered。 2026-06-26 07:19 follow-up：`gitea/main` 已包含前一輪 SOP 文件 commit `1fd5e2a8`，ArgoCD `awoooi-prod` 讀回 `Synced / Healthy`，revision `1fd5e2a8b0f18d24eed16aa2a44286bcbf230603`，API `2/2`、Web `2/2`、Worker `1/1`，pods `restart=0`。重跑 full cold-start 仍是 `PASS=87 WARN=0 BLOCKED=0`，result `GREEN`。直接 public route 讀回：AWOOOI API `200`、AWOOOI Web `307`、VibeWork `200`、AwoooGo `200`、MOMO health `200`、Stock freshness `200`、Bitan `200`、Gitea `200`、Harbor `200`、Registry `/v2/` expected `401`、Sentry expected `302`、SigNoz `200`、Langfuse `200`。188 blocker 精準分類：`pg_lsclusters` 顯示 host PostgreSQL `14/main` down，`systemctl status postgresql@14-main` 顯示 `invalid primary checkpoint record` 與 `PANIC: could not locate a valid checkpoint record`；`certbot.service` 顯示 `sentry.wooo.work` renew rate-limited，`snap.certbot.renew.service` 顯示 challenge failed；`awoooi-startup.service` 曾嘗試以 root 執行 `pg_resetwal` 並失敗。本輪不執行 `pg_resetwal`、不 `reset-failed`、不重啟 service；188 需用獨立維護窗口、rollback owner、restore/source-of-truth plan 處理，詳見 `docs/runbooks/HOST-188-HYGIENE-MAINTENANCE-RUNBOOK.md`，並可先跑 `scripts/reboot-recovery/188-host-hygiene-maintenance-checklist.sh --no-color` 取得只讀 preflight。110 load 已降到約 `4.83 / 4.82 / 5.52`，top CPU 是 active AWOOOI Web `turbo build` / Docker buildx；Swap 仍滿但 memory available 約 `41Gi`，本輪不手動清 swap。整體宣告仍是 `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。 2026-06-26 07:02 全主機 live refresh：`110 / 120 / 121 / 188 / 112 / 111 / 168` ping 與 SSH port 全部 OK。110 `systemctl=running`、failed units `0`，但 load `5.83 / 7.26 / 5.77` 且 top CPU 是 AWOOOI Web `next build`，Swap 仍 `7.8Gi / 7.8Gi`；這是 CI/build 壓力，不是 orphan Chrome 或 Docker 事故。120 / 121 `systemctl=running`、K3s active，nodes `mon` / `mon1` 均為 Ready。ArgoCD `awoooi-prod` 在 06:57 曾短暫 `OutOfSync / Progressing`，因 deploy marker `52f61da4` rollout 正在替換 API/Web/Worker；07:00 後已穩定為 `Synced / Healthy`，API `2/2`、Web `2/2`、Worker `1/1`，API/Web 仍跨 `mon` / `mon1`。重跑 live cold-start：`PASS=87 WARN=0 BLOCKED=0`，result `GREEN`。StockPlatform `/api/v1/system/freshness` 曾在容器剛重啟約 35 秒時短暫 `502`，後續連續讀回皆 `200` 且 `status=ok`、`latest_trading_date=2026-06-25`、blockers `[]`；這類 rollout warmup 只有連續失敗才算 blocker。MOMO health 是 `V10.699`，cold-start direct evidence 仍顯示 current-month parity `15383 / 15383` 截至 `2026-06-24`，daily freshness `1|2026-06-24`。Backup status 06:58：110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、offsite/rclone fresh、`last_backup_all=2026-06-26 02:31:02`、`escrow_missing=5`。188 產品容器健康，但 host `systemctl=degraded` 仍是真實 host hygiene blocker：`awoooi-startup.service`、`postgresql@14-main.service`、`certbot.service`、`snap.certbot.renew.service` failed。112 Wazuh manager/indexer/dashboard active，ports `1514 / 1515 / 55000` listen，但 production Wazuh route 仍回報 `disabled_waiting_iwooos_wazuh_owner_gate`、`configured=false`、manager registry accepted `0`、runtime gate `0`。111 / 168 可連線，但兩邊 AWOOOI dev workspaces 皆 ahead 17 且 HEAD 不同（`111=56c83257`、`168=59485d51`）；Mac Mini `/System/Volumes/Data` 只剩約 `3.2Gi`。目前 service recovery 宣告維持 `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`，host hygiene / DR escrow / Wazuh registry / workstation capacity 明確列為 service green 之外的 blocker。 2026-06-26 06:50-06:55 188 host hygiene read-only triage：188 product services remain green, but host `systemctl` is still `degraded` and must not be smoothed into full host green. Failed units are `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, and `snap.certbot.renew.service`. Evidence shows the host PostgreSQL cluster `14/main` is down in `pg_lsclusters`, while product DB / exporters still respond through containerized services; therefore `pg_isready` or `pg_up=1` cannot substitute for host cluster health. The 188 startup service detected `could not locate a valid checkpoint record` on 2026-06-23 and attempted `pg_resetwal` as root, which failed; v1.63 treats PostgreSQL checkpoint/WAL errors as break-glass only and the repo-side startup script now fails closed instead of running `pg_resetwal`. Certbot renew for `sentry.wooo.work` is also failing and hit ACME rate-limit / challenge failure, but the public cert is still valid until 2026-07-09 16:03:40 UTC. Current declaration: `SERVICE_GREEN_HOST_HYGIENE_BLOCKED` for 188, while overall service recovery remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`. 2026-06-26 06:40-06:44 全主機 read-only refresh：`110 / 120 / 121 / 188 / 112 / 111 / 168` ping 與 SSH port 全部 OK。核心 reboot scope 維持 green：110 `systemctl=running`、failed units `0`，Docker / Gitea / Harbor / Prometheus / Alertmanager 可用；120 / 121 `systemctl=running`、failed units `0`，K3s nodes `mon` / `mon1` Ready；188 產品容器與 PostgreSQL / Redis / MOMO / SignOz 可用。ArgoCD `awoooi-prod` 已從先前 degraded 收斂為 `Synced / Healthy`，revision `b2945ab9f716d9d685434ae0e67b9318414b27fe`；`km-vectorize` official 03:00 台北時間 run 成功，`lastSuccess=2026-06-25T19:00:14Z`。Public routes for AWOOOI / VibeWork / AwoooGo / MOMO / Stock / Bitan / Gitea / Harbor / Registry / Sentry / SigNoz / Langfuse return expected statuses; AWOOOI API health is `healthy / prod / mock_mode=false`; MOMO health is `V10.690`; StockPlatform freshness is `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`; backup-status remains core green with `escrow_missing=5`. Boundaries: 188 host still has failed units `awoooi-startup.service`, `certbot.service`, `postgresql@14-main.service`, `snap.certbot.renew.service` that require host hygiene cleanup; 112 Wazuh services / ports are active but Wazuh manager registry accepted remains `0`; 111 / 168 Codex workspaces are reachable but have different local HEADs on the same ahead branch; Mac Mini free space is about `3.4Gi`. Current service verdict remains `FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, not `DR_COMPLETE` or `Wazuh recovered`. 2026-06-26 06:26-06:28 隔日 read-only refresh：四主機 ping/SSH OK，cold-start `PASS=89 WARN=0 BLOCKED=0`，MOMO `V10.690` 且 latest import job `57 completed`，StockPlatform `/api/v1/system/freshness` 仍為 `status=ok` / `latest_trading_date=2026-06-25` / blockers `[]`，backup-status 110 `13/13 fresh failed=0`、188 `2/2 fresh failed=0`、`core_blockers=0`、`offsite_fresh=1`、`rclone_gdrive_fresh=1`、`last_backup_all=2026-06-26 02:31:02`、`escrow_missing=5`。06:26 full wrapper 首輪在 `https://awoooi.wooo.work/zh-TW/iwooos` 與 `https://vibework.wooo.work/` 出現單次 `000`，但獨立 curl 立即回 `200`，route-only wrapper 也回 `PASS=31 WARN=0 BLOCKED=0 RESULT=GREEN`；因此 v1.61 將 public route gate 改為最多 3 次 retry，只有連續失敗才算 `BLOCKED`，retry 後恢復則列為 evidence warning。06:28 core wrapper with routes skipped returns `POST_START_QUICK_CHECK PASS=15 WARN=2 BLOCKED=0`, `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`。本次沒有 Docker/systemd/Nginx/firewall/K8s/DB/Wazuh runtime 寫操作。 2026-06-25 21:14 StockPlatform natural-cron / full-wrapper refresh supersedes the 20:25 product-data blocker wording. After waiting for official schedules instead of manual ingestion, `intelligence-sync` 21:00 finished `status=0`, `core.margin_short_daily` reached `2026-06-25` / 1976 rows, and `ai-recommendation-pipeline` 21:10 finished `STOCKPLATFORM_AI_RECOMMENDATION_PIPELINE_OK as_of_date=2026-06-25` with `draft_count=120`, `candidate_count=120`, and `rag_documents=1000`. StockPlatform `/api/v1/system/freshness` now returns `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`, with price / chips / margin / AI recommendations all on `2026-06-25`. The 21:14 full wrapper returns cold-start `PASS=89 WARN=0 BLOCKED=0` and overall `POST_START_QUICK_CHECK PASS=38 WARN=2 BLOCKED=0`, `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`. The only remaining recovery red gate is DR credential escrow evidence `escrow_missing=5`; Wazuh manager registry accepted remains `0` as a security evidence blocker, not a reboot service blocker. 2026-07-01 13:35 110 CPU/load 判讀規則更新：`docker_stats.prom` 必須先看 mtime / freshness，超過 300 秒不得作為當下 container CPU 歸因或 blocker；若 110 `node_load5` 高但 Prometheus CPU mode 仍有 idle、`awoooi_host_gitea_actions_active_process_count=0`、orphan browser count=0，主 blocker 不得誤寫成 Gitea / Playwright / Stock smoke CPU。此時優先看 `diagnose-110-ssh-publickey-auth.sh` 的 `NODE_LOAD_CLASSIFIER`、`DOCKER_STATS_TEXTFILE_FRESHNESS` 與 `SYSTEMD_UNIT ... classifier=systemctl_show_timeout|systemctl_timeout_budget_exhausted`。外部 SSH userauth timeout 時，cold-start 必須輸出 `SSH_110_BLOCKER remote_control_channel_unavailable` 與 `SSH_110_NEXT_ACTION local_console_run_recover_110_control_path_and_harbor_local_check`；下一步是 110 本機 console / 已恢復 control path 執行 `recover-110-control-path-and-harbor-local.sh --check`，不是重跑 Harbor workflow 或用舊 docker stats 指認 Gitea。 2026-07-01 14:05 追加 controller / SLO stale-attribution guard：`host-sustained-load-controller.py` 與 `host-sustained-load-evidence.py` 必須把超過 `300s` 的 Docker stats 樣本標成 untrusted；`top_container_cpu` / `top_containers` 不得使用 stale `docker_container_cpu_cores`，舊值只能留在 `top_container_cpu_untrusted` / `top_containers_untrusted` 當證據。`reboot-auto-recovery-slo-scorecard.py` 若收到 `docker_stats.fresh=false` 或 `top_containers_fresh=false`，只能保留 `host_pressure_high_load` 與 `host_container_cpu_attribution_stale`，不得產生 `host_110_gitea_cpu_pressure`。此時下一步固定為恢復 Docker stats textfile exporter 或收集 sanitized host pressure，且仍不得重啟 Docker / Nginx / K3s / DB / firewall、不得恢復 generic runner、不得用 stale Gitea CPU 樣本取消或 drain 任何工作。 2026-07-01 23:28 追加 110 主動告警與 controlled quota 收斂；2026-07-02 00:40 更新 Gitea cap；2026-07-02 11:54 追加 process-family 主動告警：`HostLoadAverageSustainedHigh` 的 `load5/core > 1.5 for 15m` 只能當 critical 門檻，不足以回答「110 CPU 又持續偏高為何沒告警」。Prometheus 必須同時有 `Host110SustainedModeratePressure`，在 `awoooi_host_load5_per_core{host="110"} > 0.75`、Gitea / StockPlatform 關鍵容器 `docker_container_cpu_cores > 1.0`，或 `awoooi_host_process_family_cpu_percent{family=~"systemd_control_plane|ssh_control_plane|gitea_service|postgres"} > 50` 持續 1 分鐘時告警，auto-repair action 必須指向 110 已部署的 `/home/wooo/scripts/host-sustained-load-controller.py --load5-per-core-threshold 0.75 --hot-container-cpu-threshold 1.0 --process-family-cpu-threshold 50`。`host-runaway-process-exporter.py` 必須輸出 sanitized process-family metrics，只允許 family / comm labels，不得輸出 raw command line、workspace path、URL 或 secret。Gitea runtime CPU quota 已從 `docker update --cpus=2 gitea` 進一步收斂為 `docker update --cpus=1.5 gitea`；rollback 只允許暫回 `docker update --cpus=2 gitea` 並需 Prometheus readback，不可重啟 Docker daemon 或 Gitea container。若 `BackupAggregateRunFailed` 只因 `exported_job="backup_all"` 舊 aggregate failed_count firing，而 component jobs / `backup-status.sh` 已 green，必須視為噪音並用 component failed count 判斷，不得讓它干擾 cold-start / Telegram 主線。 2026-07-02 12:15 追加 controller command path 合約：`host-sustained-load-controller.py` 產出的 `dry_run_command` / `post_apply_verifier` 必須使用 110 實際部署路徑 `/home/wooo/scripts/host-sustained-load-evidence.py`、`/home/wooo/scripts/host-sustained-load-controller.py`、`/home/wooo/scripts/host-runaway-process-remediation.py`。若 controller 在 live host 上輸出 `scripts/ops/...` 這類 repo 相對路徑，視為告警自動化斷鏈，需先修 controller / `--script-dir`，再進入 playbook check-mode；不得把「helper 找不到」當成 CPU 根因已處理。 2026-07-02 12:35 追加 110 CPU 分流優先序：若 Docker stats 是 fresh，且 `gitea` 或 StockPlatform 關鍵容器已超過 early triage 門檻 `1.0 core`，controller / evidence 必須先路由到對應服務 playbook，不得被長壽命 `ps %CPU` 的 `systemd_control_plane` 平均值搶先導到 control-plane playbook。control-plane saturation 仍保留為後備路徑，適用於沒有已知 hot container / hot service family 的情境。 2026-07-02 12:50 追加 Gitea 壓力 check-mode：`host-sustained-load-controller.py` 若分類為 `blocked_gitea_queue_or_hook_backlog_requires_playbook`，`dry_run_command` 必須指向 110 已部署的 `/home/wooo/scripts/gitea-queue-hook-backlog-playbook.py --host 110 --metrics-file /home/wooo/node_exporter_textfiles/host_runaway_process.prom --docker-stats-file /home/wooo/node_exporter_textfiles/docker_stats.prom --json`。此 playbook 只能讀公開 Gitea health/version/metrics 與 sanitized host textfile；必須輸出 health、version、active Actions、fresh Docker stats、`gitea_hooktasks`、Gitea container CPU、process-family CPU 與 operation boundaries。若 health/version 正常、active Actions 為 0、`gitea` container 超過 `1.0 core` 且 `gitea_hooktasks >= 1000`，狀態固定為 `blocked_gitea_hooktask_backlog_check_required`，下一步走 public queue / hook backlog 判讀；不得讀 app.ini / secret / `.runner` / raw session / DB，不得重啟 Gitea / Docker / Nginx / K3s / DB / firewall。 2026-06-25 20:25 orphan Chrome cleanup / scorecard refresh supersedes the 20:11 CPU wording. 110 high CPU was traced to two `stockplatform-review-bulk-ux` Chrome process groups `2756503` and `2829627` with root Chrome process `PPID=1`, elapsed about 5h, no active parent smoke, and sustained GPU/renderer CPU. With user approval, only those two process groups received targeted `SIGTERM` at 20:24. Post-check showed no remaining PGID entries; `vmstat` showed CPU idle around `85-90%`, `si/so=0`, and no immediate swap thrash. No Docker/systemd/Nginx/firewall/K8s action, CI cancellation, manual data ingestion, manual DB write, Wazuh/SOC runtime change, or secret read was performed. The 20:25 full post-start wrapper then returned cold-start `PASS=89 WARN=0 BLOCKED=0`, but overall `POST_START_QUICK_CHECK PASS=37 WARN=2 BLOCKED=1`, `RESULT=BLOCKED`, because StockPlatform data freshness was still blocked at that time and DR remained incomplete. 2026-06-25 20:11 StockPlatform cron-source recovery supersedes the 19:35 source-version wording. StockPlatform Gitea `main` and live `/home/wooo/stockplatform-v2` are now at `fb91aa4c6272469d1d26e0820169629eac17d28a fix(ops): restore production cron recovery entrypoints`; six missing production cron entrypoint scripts are restored, `run-intelligence-sync.sh` contains the Docker-backed `psql` shim, and live contract check confirms every `scripts/ops/*.sh` referenced by `install-production-cron.sh` exists. The only live write performed for StockPlatform recovery was a fast-forward `git pull --ff-only origin main` on 110; no Docker/systemd/Nginx/firewall/K8s restart, manual ingestion run, manual DB write, or secret read was performed. Natural cron evidence after the pull is now green for the repaired entrypoints: `source-remediation-queue` 19:56 and 20:00 succeeded, `market-index-ingestion` 20:00 succeeded, `price-ingestion` 20:02 succeeded, `margin-short-ingestion` 20:05 succeeded, `chips-ingestion` 20:06 succeeded, and `ai-recommendation-pipeline` 20:10 ran but correctly produced the internal blocker `core_margin_short_daily_incomplete,official_margin_short_daily_official_pending`. StockPlatform `/api/v1/system/freshness` therefore still returns `status=blocked` because the 2026-06-25 official margin-short source is pending and `ai.recommendations` must stay on 2026-06-24 until that gate clears. This is no longer a route, source-version, or missing-cron-script blocker; it is a product-data freshness blocker waiting on official source availability and the next valid AI pipeline run. 2026-06-25 19:35 product-version / data-freshness refresh supersedes the 19:06 data-complete wording. Host boot, K3s, AWOOOI runtime, MOMO service/data, backup/offsite, Bitan cleanliness, and expanded public routes are available, but the stricter post-start wrapper now checks StockPlatform `/api/v1/system/freshness` and correctly returns `RESULT=BLOCKED` when product data is not current. The 19:35 lightweight wrapper run used `--skip-cold-start --skip-backup --skip-cpu` after the 19:24 full host/cold-start/backup readback and returned `PASS=31 WARN=1 BLOCKED=1`, with the single blocker `StockPlatform freshness is blocked: core_margin_short_daily_missing,ai_recommendations_stale`. `stock.wooo.work`, `/healthz`, and `/api/healthz` all return `200`; public routes now covered by the wrapper include AWOOOI, VibeWork, AwoooGo, 2026FIFA, Agent Bounty, MOMO, Stock, Bitan, TsenYang, VTuber, Gitea, Harbor, Registry, Sentry, SigNoz, Langfuse, and AIOps. Do not declare "all products and data are latest" until StockPlatform freshness is `ok`; keep DR blocked until `escrow_missing=0`. 2026-06-25 19:06 post-CD live read-only refresh supersedes the 18:53 wrapper wording. Consecutive main pushes caused older deploy markers to be replaced, so the latest production truth is deploy marker `d8ca8224 chore(cd): deploy 9dbe044 [skip ci]`. Read-only ArgoCD shows `awoooi-prod Synced / Healthy` at revision `d8ca822422021d0fda8da8fa4c354c0c4db7ff22`; API/Web/Worker live image tag `9dbe044ea1e8e3894ccbeb5ed760bb124b87f7be`; API `2/2`, Web `2/2`, Worker `1/1`. The 19:05 post-start quick check returns `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, delegated cold-start remains `PASS=89 WARN=0 BLOCKED=0`, and 19:05-19:06 route stability checks confirm AWOOOI API, IwoooS, AwoooGo, Stock, VibeWork, Bitan, and MOMO health all return `200` for five consecutive external reads. Earlier AwoooGo / Stock `502` reads were post-deploy upstream warmup transients, not persistent service failures. Hosts, routes, K3s, AWOOOI API health, MOMO service health, MOMO business data freshness, backup core/offsite, and core monitoring/exporter surfaces are green for controlled runner/CD release. MOMO is healthy on `V10.690`; latest import job `57` completed cleanly; `MOMO_DAILY_FRESHNESS 1|2026-06-24`; current-month daily snapshot and realtime tables match through `2026-06-24`. `post-start-quick-check.sh` parses cold-start `PASS / WARN / BLOCKED` summary before classifying exit codes, so WARN-only rollout/stale evidence is no longer inflated into a service blocker. The wrapper returns `RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED` when service blockers are zero but `escrow_missing=5` remains. Do not turn this into a DR complete or security/runtime acceptance claim. Wazuh production routes are now `200 disabled_waiting_iwooos_wazuh_owner_gate`, but `configured=false`, manager query accepted `0`, manager registry accepted `0`, and runtime gate `0`; treat Wazuh as a security registry evidence blocker, not a reboot service blocker. ```text Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%. Live cold-start read-only check: 2026-06-25 19:05 wrapper delegated cold-start PASS=89 WARN=0 BLOCKED=0, Result=GREEN. Post-start quick check: 2026-06-25 21:14 PASS=38 WARN=2 BLOCKED=0; warning split SERVICE=0 BOUNDARY=1 EVIDENCE=1; Result=FULL_STACK_GREEN_DR_ESCROW_BLOCKED. Cold-start layer remains GREEN and StockPlatform freshness is now OK; DR remains blocked by credential escrow evidence. Repo-side cold-start v1.42+ live read-only run: MOMO source absence / stale data blocker is cleared by import job 57 and `MOMO_DAILY_FRESHNESS 1|2026-06-24`. Live 110 script sync is not claimed until a separate approved deployment/sync happens. 110 live-sync parity: 2026-06-24 23:15 read-only `verify-cold-start-monitor-deploy.sh` correctly BLOCKED because repo script hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05` differs from 110 live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`. Do not use live 110 monitor output to prove v1.42 behavior until the approved live-sync gate in §13.3.1 passes. Service state: FULL_STACK_GREEN_DR_ESCROW_BLOCKED; 110/120/121/188 reachable, K3s mon/mon1 Ready, public routes/TLS green, MOMO data fresh, StockPlatform data fresh, 110/188 backup health fresh, 188 node-exporter / PostgreSQL exporter / Redis exporter restored, 188 MinIO endpoint and Velero BackupStorageLocation restored, 110 disk pressure cleared, and 110 orphan StockPlatform Chrome smoke groups cleared by targeted approved SIGTERM. StockPlatform production cron source drift is repaired and verified by natural cron runs; product-data completeness is now green for the 2026-06-25 evidence set. Runtime release state: API/Web/Worker live image tag is `9dbe044ea1e8e3894ccbeb5ed760bb124b87f7be`, and 19:06 K3s readback shows API/Web/Worker pods Running; production API health returns healthy with `environment=prod`, `mock_mode=false`, and postgresql / redis / openclaw / signoz / gcp ollama providers up. 19:05 route smoke returned 200 for AWOOOI API, IwoooS, MOMO health, and Stock; cold-start route gate also returned expected statuses for AWOOOI web, MOMO, Gitea, Harbor, Registry, Sentry, SigNoz, Langfuse, Bitan, and AIOps. AwoooGo, Stock, AWOOOI API, IwoooS, VibeWork, MOMO health, and Bitan then returned 200 for five consecutive external route reads from 19:05:38 to 19:06:24. 19:35 expanded route readback returned expected 2xx/3xx for AWOOOI, VibeWork, AwoooGo, 2026FIFA, Agent Bounty, MOMO, Stock, Bitan, TsenYang, VTuber, Gitea, Harbor, Registry, Sentry, SigNoz, Langfuse, and AIOps. Cold-start raw route gate returned all expected route statuses, including redirects such as awoooi web=307 and sentry=302. MOMO release state: mo.wooo.work health is healthy on version V10.690. `momo-pro-system`, `momo-scheduler`, and `momo-telegram-bot` are healthy; scheduler `RestartCount=0`. 18:23 dedicated preflight returns PASS=19 WARN=2 BLOCKED=0, so retain recent container-replace / scheduler fail-closed / notification evidence notes, but no service blocker remains. MOMO data state: current-month daily_sales_snapshot and realtime_sales_monthly match through 2026-06-24: `daily_sales_snapshot=109061|2025-07-01|2026-06-24`, `MOMO_MONTHLY_SYNC 15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`, and `MOMO_DAILY_FRESHNESS 1|2026-06-24`. Latest import job is `57 completed|即時業績_當日.xlsx|2026-06-25T13:16:47.359958|2026-06-25T13:18:02.964985|15383|15383|0`. StockPlatform data state: `/api/v1/system/freshness` returns `status=ok`, `latest_trading_date=2026-06-25`, blockers `[]`. Current OK sources include `core.price_daily` 2026-06-25 / 1976 rows, `core.chips_daily` 2026-06-25 / 1976 rows, `core.margin_short_daily` 2026-06-25 / 1976 rows, `core.market_index_daily.tw` 2026-06-25 / 2 rows, `core.market_index_daily.global` 2026-06-25 / 2001 rows, and `ai.recommendations` 2026-06-25 / 2868 rows. The 21:10 natural AI pipeline produced `STOCKPLATFORM_AI_RECOMMENDATION_PIPELINE_OK as_of_date=2026-06-25`; no manual ingestion or DB write was performed. Product version readback: StockPlatform live source `/home/wooo/stockplatform-v2` matches Gitea `wooo/stockplatform-v2.git` main `fb91aa4c6272469d1d26e0820169629eac17d28a`; VibeWork live image `192.168.0.110:5000/vibework/web:76a4ee15026af278a3660ad4b4547e9308b107be` matches Gitea `wooo/vibework.git` main `76a4ee15026af278a3660ad4b4547e9308b107be`; AwoooGo live source `/home/wooo/awooogo` matches Gitea `wooo/AwoooGo` main `6897972e9820cbb96c508fa9a80c66246c973307`; MOMO runtime uses `registry.wooo.work/wooo/momo-pro-system:stable` image id `df931906e158` created `2026-06-25T13:28:59+08:00`, while Gitea `wooo/momo-pro-system.git` main is `25120cbf21ba51affc94d0220ec87e607f59a833`; 188 runtime directory is a compose/image deployment path, not a git worktree, so add image revision label evidence before declaring code-image parity. Google Drive / source-file state: 14:16 cold-start reports `MOMO_GDRIVE_TOKEN_STAT 100000:100000:600 scheduler_uid=100000`. Dedicated preflight confirms host token metadata matches scheduler UID and restrictive mode; container token artifact exists with mode `600`. Token content was not read. Future Drive auth/API failure must still be treated as failed import evidence rather than no-file success. 110 CPU/load readback: 2026-06-25 10:58 user-approved minimal SIGTERM targeted only orphan `stockplatform-review-bulk-ux` Chrome process groups `438005`, `471295`, `640155`, and `670628`; `OLD_GROUPS_REMAINING` returned empty. 20:24 readback found a second recurrence with orphan process groups `2756503` and `2829627`, root Chrome `PPID=1`, elapsed about 5h, no active parent smoke, GPU process CPU around 96%, and renderer CPU around 22%; approved targeted `SIGTERM` cleared both PGIDs. 21:14 CPU attribution shows current load is dominated by an active AWOOOI Web `next build` process group and its worker processes, not orphan Chrome. No Docker/systemd/Nginx/firewall/K8s write was performed; do not cancel active CI/smoke unless separately approved. If Chrome groups are active children of Playwright / CI, observe queue and timeout; if they become PPID 1 orphan process groups with sustained CPU and no parent smoke, run dry-run and require owner approval before targeted `SIGTERM`. Backup / monitoring state: 19:05 wrapper readback confirms backup core blockers are 0, 110 is 13/13 fresh failed=0, 188 is 2/2 fresh failed=0, offsite_fresh=1, rclone_gdrive_fresh=1, integrity_stale=0, last aggregate is 2026-06-25 02:35:09, and escrow_missing=5. Route transient handling: post-deploy `502` on Stock or AwoooGo is a blocker only if it persists after upstream container health is ready and 3-5 consecutive external route reads still fail. For AwoooGo, live upstream is on 110 `192.168.0.110:32190`; do not test only `127.0.0.1` on 110 because the listener may bind the host address. For K3s workload balancing, wait for terminating pods to disappear before judging API/Web placement; final required state for two-replica API/Web is split across `mon` and `mon1`. Notification-noise state: healthy AWOOOI heartbeat is suppressed; heartbeat warning dedupe uses stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes; MOMO Pro monitor uses https://mo.wooo.work/health as primary truth and no longer checks the 188 root path; MoWoooWorkDown now labels component=momo-pro-system and requires public/local/container/data-freshness triage instead of blind restart; docker-health-monitor keeps 5-minute repair cadence but has a separate 30-minute Telegram fallback cooldown; Bitan public-content check keeps failure alerting with same-fingerprint cooldown and one recovery notice. Deploy storm / CD replacement state: if several main commits land during recovery, older CD runs may be canceled by newer commits. Do not treat the canceled run as a service failure. Wait for the final deploy marker, verify live image tags, ArgoCD health, public routes, DB freshness, backup status, and post-start quick check before declaring latest production recovered. Wazuh / SOC boundary state: production Wazuh read-only route presence is not equivalent to Wazuh registry recovery. `/api/iwooos/wazuh` and `/api/v1/iwooos/wazuh` returning `200 disabled_waiting_iwooos_wazuh_owner_gate` only proves the route boundary is deployed; manager registry accepted, owner evidence accepted, active response, host write, agent re-enroll, restart, secret patch, Kali active scan, and runtime gate remain `0 / false`. Monitoring coverage recovery state: if CD post-deploy fails only because `scripts/generate_monitoring.py --check` reports `nginx-exporter` down on `192.168.0.188:9113`, first verify 188 `stub_status` and restore the stateless exporter with `scripts/ops/188-nginx-exporter-restore.sh`; do not reload Nginx or restart product containers for this symptom. Allowed declaration: host boot, core service readiness, K3s, public route availability, AWOOOI API health, MOMO service health/data freshness, Bitan public-content cleanliness, and backup/offsite readiness are green for the latest read-only evidence set. Forbidden declaration: all product data latest, StockPlatform data freshness green, DR complete, credential escrow complete, Wazuh host registry accepted, 110 live monitor synced, or runtime/security acceptance. Credential escrow evidence is still missing and StockPlatform freshness is blocked; neither may be smoothed into green. ``` 2026-06-24 22:17 Codex workstation continuity readback: ```text MacBook Pro 192.168.0.111 can authenticate to Gitea over SSH with its own public key named MacBook Pro Codex 20260624. MOMO Pro Mac Mini workspace is /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty=0. MOMO Pro MacBook workspace is /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty=0. MOMO import-boundary regression: pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q => 10 passed. MOMO production release: Gitea main and cd.yaml #904 are at 84035906aba0e5e190d031a13cfd9b47a8cd1f73; 188 live source marker proves production deploy. Codex Start Here / workstation dashboard / scorecard safe artifacts were copied to MacBook Pro; latest artifact dashboard readback is refreshed after the docs closeout commit. Raw Codex App DB, auth, sessions, raw conversations, .env, runtime volumes, raw .git directories, passwords, tokens, and Mac Mini private keys were not copied. AwoooGo MacBook dev workspace remains ready at /Users/ooo/codex-workspaces/awooogo-dev, branch dev, upstream gitea/dev, commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty=0. Safe handoff artifacts still match by local / remote SHA-256 readback after Start Here / workstation dashboard / scorecard refresh. Exact hash values are intentionally not hard-coded in this runbook because they change whenever handoff artifacts are refreshed. Raw Codex App DB, auth, sessions, raw conversations, .env, runtime volumes, raw .git directories, passwords, tokens, and Mac Mini private keys were not copied. This improves workstation continuity after host reboot / operator relocation, and the MOMO import-boundary fix is now production-deployed; it does not change service cold-start status: full-stack green remains blocked by MOMO data freshness and DR remains blocked by credential escrow evidence. ``` 2026-06-18 12:17 live readback supersedes older service-availability wording: ```text Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%. Live cold-start read-only check: PASS=83 WARN=1 BLOCKED=0, Result=DEGRADED. Service state: SERVICE_AVAILABLE_DEGRADED; 110/120/121/188 reachable, K3s mon/mon1 Ready, NODE_FS_ERROR_EVENTS=0, public routes/TLS green, 110/188 backup health fresh. Rollout state after transient 12:14 startup window: awoooi-api 2/2, awoooi-web 2/2, worker 1/1, canary 1/1, public API health 200 healthy. Only live warning: retained stale K8s Job km-vectorize-29689620 from 2026-06-14 03:00. Later official km-vectorize Jobs 29692500 / 29693940 / 29695380 are Complete. Allowed declaration: services are available with one stale failed Job warning. Forbidden declaration: full cold-start green, DR complete, or runtime/security acceptance. ``` 2026-06-18 13:43 live readback supersedes the stale-Job warning wording: ```text Repo-side reboot SOP / Plan B / automation contracts: COMPLETE, 100%. Live cold-start read-only check: PASS=84 WARN=0 BLOCKED=0, Result=GREEN. Service state: FULL_STACK_GREEN_FOR_SERVICE; 110/120/121/188 reachable, K3s mon/mon1 Ready, NODE_FS_ERROR_EVENTS=0, public routes/TLS green, 110/188 backup health fresh. K8s Job classification: FAILED_JOBS=1, STALE_FAILED_JOBS=1, ACTIVE_FAILED_JOBS=0. The retained km-vectorize failure stays as evidence but no longer blocks service readiness after later official successful Jobs. Allowed declaration: full cold-start service readiness is green for this evidence set. Forbidden declaration: DR complete or runtime/security acceptance. Credential escrow evidence is still missing and must not be forged. ``` 2026-06-18 14:31 live runaway-process readback supersedes repo-only AIOps wording: ```text 110 host runaway process exporter is live-installed and scraped. Textfile source: /home/wooo/node_exporter_textfiles/host_runaway_process.prom. Prometheus readback: monitor_up=1, orphan_browser_groups=0 for headless_browser_smoke and stockplatform_headless_smoke, active Gitea Actions containers=2, load5_per_core around 0.79-0.81, swap_used_ratio around 1.0, remediation_authorized=0. Alerts: HostRunawayProcessMonitorMissing is not firing; HostOrphanBrowserSmokeHighCpu is not firing. Allowed declaration: runaway Chrome/smoke recurrence guard is live and scraped. Forbidden declaration: AI runtime remediation is enabled. Remediation remains gated and must not execute without owner approval, maintenance window, evidence ref, dry-run, and post-check. ``` 2026-06-18 14:51 production event-packet readback: ```text Host runaway alert-to-event packet is deployed in production. Deploy marker: 2d278568 chore(cd): deploy f358a0f [skip ci]. Runtime image: awoooi-api / awoooi-web / awoooi-worker use f358a0f6c3e614e407dedb6eee89bf10b2bc8173. ArgoCD readback: awoooi-prod Synced / Healthy. Alert mapping: HostOrphanBrowserSmokeHighCpu -> orphan_browser_smoke_runaway_process; HostCiRunnerLoadSaturation -> ci_runner_load_saturation. Allowed declaration: monitoring, alert rules, live scrape, AI event packet routing, PlayBook / KM contract, and production deployment are complete for this evidence set. Forbidden declaration: Telegram send, Bot API call, Gateway queue write, process kill, Docker/systemd restart, Nginx reload, firewall/K8s action, or runtime remediation is authorized. ``` 2026-06-18 16:08 P3-009 Host Runaway AIOps product readback: ```text Host runaway AIOps closed-loop read model is deployed in production. Deploy marker: 42c08ece chore(cd): deploy 27143fb [skip ci]. API endpoint: /api/v1/agents/agent-host-runaway-aiops-loop-readiness. Production readback: schema_version=host_runaway_aiops_loop_readiness_v1, current_task_id=P3-009, next_task_id=P3-010, completion=100, loop_stage_count=6, alert_lane_count=2, asset_writeback_contract_count=5. Host 110 live readback in the model: orphan browser groups=0, active CI containers=2, remediation_authorized=0, runtime/write counters=0. Governance route: /zh-TW/governance?tab=automation-inventory shows P3-009 on desktop 1440x1100 and mobile 390x844 with missing text=0, console/page errors=0, horizontal overflow=false. Allowed declaration: monitoring, alert rules, AI event packet, PlayBook / KM contract, Verifier/writeback contract, gated remediation dry-run boundary, and product-visible readback are complete for this evidence set. Forbidden declaration: AI runtime remediation is enabled. Process termination, Docker/systemd restart, Nginx reload, firewall/K8s action, Telegram live send, Gateway queue write, Bot API call, production write, and secret read remain forbidden without owner approval, maintenance window, evidence ref, dry-run, and post-check. ``` | 項目 | 2026-06-24 11:35 Asia/Taipei live result | 判定 | |------|-------------------------------------------|------| | Overall recovery readiness | `98%` | `SERVICE_AVAILABLE_MOMO_SOURCE_BLOCKED_DR_ESCROW_BLOCKED` | | P0 host / K3s recovery | `100%` | `DONE` | | P1 backup / alert / escrow | `96%` | `BLOCKED_DR_ESCROW` | | P2 service / data truth | `96%` | `BLOCKED_MOMO_DATA_FRESHNESS` | | P3 docs / automation contracts | `100%` | `DONE_WITH_MOMO_SOURCE_ABSENCE_GATE_V142_REPO_ONLY` | | 110 host runtime | `fwupd-refresh.timer` intentionally disabled/inactive after non-runtime firmware metadata refresh failed units were classified; `systemctl --failed` returns `0 loaded units listed`; rollback is `sudo systemctl enable --now fwupd-refresh.timer` | `GREEN_WITH_FWUPD_TIMER_DISABLED` | | 110 host runaway process guard | 14:31-14:32 live scrape confirms `monitor_up=1`, orphan browser groups `0`, active Gitea Actions containers `2`, `load5_per_core≈0.79-0.81`, `swap_used_ratio≈1.0`, and `remediation_authorized=0`; exporter/helper also remain in Ansible textfile exporter source-of-truth. | `LIVE_SCRAPED_RUNTIME_GATE_0` | | 120 reachability | ping OK, SSH OK, boot around `2026-06-14 02:23`, K3s active, node `mon Ready` | `GREEN` | | 121 reachability | ping OK, SSH OK, failed units `0` | `GREEN` | | 188 host runtime | production services green, but host `systemctl` degraded by `awoooi-startup.service`, `postgresql@14-main.service`, `certbot.service`, and `snap.certbot.renew.service`; host PostgreSQL cluster `14/main` is down while product DB containers/exporters are healthy; certbot renewal for shared `sentry.wooo.work` certificate is failing but public cert is still valid until 2026-07-09 UTC | `SERVICE_GREEN_HOST_HYGIENE_BLOCKED` | | K3s node state | `mon Ready control-plane`, `mon1 Ready control-plane`; bad pods `0`; `FAILED_JOBS=1`, `STALE_FAILED_JOBS=1`, `ACTIVE_FAILED_JOBS=0` | `GREEN_WITH_RETAINED_EVIDENCE` | | 110 -> 120 / 188 SSH trust | 00:33 cold-start exposed stale `known_hosts`; backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; final repair backup `/home/wooo/.ssh/known_hosts.before-120-188-final-refresh.20260613-011949`; CD fix `80e6ec1a` moves deploy trust to `/home/wooo/.ssh/deploy_known_hosts`; 01:28 global `known_hosts` still contains 120 / 188 and was not clobbered by deploy marker `e4a349bc` | `GREEN_WITH_GUARDRAIL` | | Backup status | 11:20 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`; escrow readback still shows `ESCROW_MISSING_COUNT=5` | `GREEN_WITH_DR_ESCROW_WARNING` | | Offsite sync / verify | 01:28 textfile: `awoooi_backup_offsite_remote_verify_ok=1`, `full_verify_fresh=1`, all 13 repos have `snapshot_count=1` and `snapshot_latest_only=1`; latest scheduled verifier log is 2026-06-12 07:20 | `GREEN` | | Backup / cold-start alerts | 01:27 live visibility check confirms Prometheus and Alertmanager expose the 5 required credential escrow gap alerts; Prometheus rules API has all five required alert names healthy; label contract check loads 24 baseline backup alert rules | `GREEN_WITH_EXPECTED_REDLIGHTS` | | Cold-start scorecard | 11:35 read-only scorecard：`PASS=86 WARN=0 BLOCKED=1`。Public routes / TLS、momo DB parity、backup exporters、120/121 K3s、MinIO / Velero、AWOOOI API/Web 皆通過；only blocker is MOMO data freshness. | `BLOCKED_MOMO_DATA_FRESHNESS` | | momo DB parity | `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17` | `GREEN` | | momo scheduler | container healthy; Drive listing from container works; pending folder `當日業績匯入` count is `0` for `即時業績_當日`; no current `Permission denied` evidence in the latest readback | `GREEN_WITH_SOURCE_ABSENT` | | ArgoCD app health | 11:35 readback: `awoooi-prod` sync `Synced`, health `Healthy`, source revision `7db7800e399caed5487a705c81ec993dec76c70f`; API/Web/Worker ready. | `GREEN` | | Workload balancing | Live API/Web/Worker/CronJob image is `e999c16b3435f197b78fe2adfeec1c4faa6c4675`; API/Web pods remain split across `mon` / `mon1`, Worker single replica remains healthy on `mon` | `GREEN` | | Credential escrow | 5 non-secret evidence markers missing | `BLOCKED` | Release rule: ```text Do not declare full cold-start green unless the latest scorecard has `WARN=0` and `BLOCKED=0`. Do not declare aggregate backup green unless latest `backup-status` has `core_blockers=0`. Do not declare DR scorecard complete while credential escrow markers are missing. ``` 2026-06-14 18:15 live rule: ```text 110 / 120 / 121 / 188 core service recovery remains available, but the latest 18:15 scorecard is DEGRADED because `WARN=1`. GO for controlled runner/CD release; keep AI auto-remediation governed by normal gates. NO-GO for "DR complete" while credential escrow evidence markers are missing. Do not fake or silence credential escrow alerts; they are the remaining correct DR red light. GO for "AWOOOI core workload balanced"; topology spread is in Gitea main / ArgoCD live and API/Web placement proves max skew <= 1. NO-GO for "full cold-start green" until `km-vectorize` failed Job is cleared by an official successful run. NO-GO for "ArgoCD fully healthy" until `km-vectorize` updates `lastSuccessfulTime` after an official scheduled Job, not a manual `UnexpectedJob`. NO-GO for any CD workflow that writes deploy host keys into `/home/wooo/.ssh/known_hosts`; deploy jobs must use an isolated `UserKnownHostsFile=/home/wooo/.ssh/deploy_known_hosts`. Current allowed wording: "core service and backup are available; 110 failed units are cleared after intentionally disabling `fwupd-refresh.timer`; high-value config Owner Packet 前台同步後 recovery readback shows no service regression; cold-start is degraded only by the `km-vectorize` official Job failure; DR complete still blocked by credential escrow; `km-vectorize` failed Job is retained but failed Pod/log are currently absent, so the next official 03:00 run remains the evidence gate." ``` 2026-06-18 12:17 live rule: ```text GO for controlled service availability: PASS=83 WARN=1 BLOCKED=0, public routes/TLS green, API health 200 healthy, API/Web/Worker/Canary ready after rollout convergence. GO for repo-side reboot readiness mechanism: readiness audit PASS=185 WARN=1 BLOCKED=0; only skipped live gate warning before the live check was run. NO-GO for "full cold-start green" until the retained stale failed Job evidence is either cleared by normal K8s history policy or explicitly accepted by an owner-provided readback package. NO-GO for "DR complete" while credential escrow evidence markers remain missing. Do not delete the failed Job manually during routine SOP verification. Keep it as evidence unless an approved maintenance window explicitly authorizes cleanup. Current allowed wording: "SOP / Plan B / automation contracts are complete; live services are available with one retained stale km-vectorize failed Job warning; hard blockers are zero; DR remains blocked by credential escrow evidence." ``` 2026-06-18 13:43 live rule: ```text GO for full cold-start service readiness for this evidence set: PASS=84 WARN=0 BLOCKED=0. GO for controlled runner/CD release under the normal security gates; this is not a bypass for owner response, runtime writer, Telegram, Gateway, K8s, Docker, Nginx, firewall, or secret operations. GO for retaining stale failed Job evidence: FAILED_JOBS=1 and STALE_FAILED_JOBS=1 are allowed when ACTIVE_FAILED_JOBS=0 and later official successful Jobs exist. NO-GO for DR complete while credential escrow evidence markers remain missing: ESCROW_MISSING_COUNT=5. NO-GO for deleting retained failed Jobs during routine verification. Cleanup requires an explicit maintenance window and owner acceptance. Current allowed wording: "full-stack service recovery is green for the current evidence set; stale km-vectorize failure is retained as historical evidence, not an active blocker; DR complete remains blocked by credential escrow evidence." ``` After any future 120 recovery, rerun this exact chain from 110: ```bash /backup/scripts/backup-configs.sh /backup/scripts/backup-all.sh /backup/scripts/sync-offsite-backups.sh --mode sync /backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color /home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1 ``` ## 0.1 When To Use This Use this SOP when any of these happen: - 110/120/121/188 reboot unexpectedly. - All services are abnormal after a power/network event. - K3s is stuck `activating`. - Host load remains high during startup and service health is mixed. - Monitoring, alerting, CD, AI auto-repair, and Docker Compose services disagree about the real state. The rule is simple: **recover the dependency chain, not the loudest symptom.** ## 0.2 啟動判定分層 重啟後不能只用一個訊號宣稱完成。每台主機與整個平台都必須分四層判定： | 層級 | 代表意義 | 最低證據 | 不代表 | |------|----------|----------|--------| | `HOST_POWERED` | 主機或 VM 看起來已通電 | console / hypervisor 顯示 running，或 LAN ARP 開始出現 | OS 已完成開機 | | `HOST_BOOTED` | OS 已進入可互動狀態 | ping OK、SSH port open、`who -b` 有本次 boot time | systemd / Docker / K3s 已健康 | | `HOST_READY` | 主機基礎服務可承接下一層 | `systemctl is-system-running` 非 degraded；failed units 可解釋；cron / docker / DB / K3s 依角色正常 | public route 或業務資料已正常 | | `SERVICE_READY` | 主機承載服務可用 | 服務 health、port、container health、DB / Redis / K3s / Harbor / Alertmanager checks 通過 | 備份、排程、告警、資料一致性與資料新鮮度已驗證 | | `FULL_STACK_GREEN` | 可以宣稱重啟恢復完成 | cold-start scorecard `WARN=0`、`BLOCKED=0`，備份/offsite/DB/告警/排程/資料新鮮度都綠 | 120 不可達或 MOMO 業務資料 stale 時永遠不能宣稱 | 2026-06-12 的 110/120 事故收斂判定是： ```text 110 HOST_READY = yes 120 HOST_READY = yes Core public services SERVICE_READY = yes FULL_STACK_GREEN = yes, because cold-start scorecard is PASS=83 WARN=0 BLOCKED=0 DR_COMPLETE = no, because credential escrow evidence is incomplete ``` 2026-06-24 的 MOMO 資料停更判定是： ```text 110 / 120 / 121 / 188 HOST_READY = yes Core public services SERVICE_READY = yes MOMO_RELEASE_CURRENT = yes, because mo.wooo.work health is V10.653 and Gitea main / CD #904 deployed commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73 MOMO_DB_PARITY = yes MOMO_DATA_FRESH = no, because latest daily_sales_snapshot date is 2026-06-17 and stale age is 7 days as of 2026-06-24 22:40 MOMO_SOURCE_AVAILABLE = no, because Drive intake 當日業績匯入 has no newer 即時業績_當日 source, scheduler stats show repeated file_count=0 runs, and Mac Mini / MacBook candidate files only contain old or header-only data FULL_STACK_GREEN = no, because live cold-start scorecard is PASS=86 WARN=0 BLOCKED=1 and repo-side v1.42 dry-run is PASS=88 WARN=0 BLOCKED=1 with blocker "188 momo source file absent while daily sales data stale" DR_COMPLETE = no, because credential escrow evidence is incomplete ``` MOMO source absence recovery gate: ```text GO: declare MOMO service recovered when health is healthy, containers are healthy, scheduler runs, DB parity matches, and release version matches Gitea/CD. NO-GO: declare MOMO data current while Drive intake has no newer 即時業績_當日 source file and latest DB bounds stop at 2026-06-17. NO-GO: re-import stale local samples, product catalog exports, header-only sheets, or already imported archive files to fake freshness. NO-GO: truncate, whole-DB restore, manual Drive movement, or manual import without explicit maintenance approval. UNBLOCK: new legitimate PChome daily-sales source appears in 當日業績匯入 or an owner-approved safe import path; import job succeeds with sync_success=true; source file moves only after success; daily_sales_snapshot and realtime_sales_monthly bounds match; MOMO_DAILY_FRESHNESS <= 2. ``` 2026-06-27 起，若已有 `momo-drive-token-source-recovery-preflight.sh` log，先跑 `python3 scripts/reboot-recovery/momo-source-arrival-gate.py --preflight-log ` 做機器判讀：`blocked_source_absent_fail_closed` 代表繼續等合法來源；`source_arrived_ready_for_safe_import_preflight` 只代表可進另一個 safe import preflight，不代表 DB write、Drive move、manual import 或 runtime write 已授權；`freshness_already_green_recheck_cold_start` 仍必須重跑同一 evidence chain 的 post-reboot summary 後才能更新恢復宣告。 所有回報必須使用這組詞，避免把「服務面可用」誤報成「整體 DR 完成」。 ### 0.3 Codex 工作站交接判定 重啟後若需要從 Mac Mini / MacBook Pro 繼續 Codex 開發，必須另外確認 Codex safe handoff artifacts，不得把服務恢復與 Codex raw 對話同步混為一談。 2026-06-24 22:17 Asia/Taipei readback： ```text MacBook Pro 192.168.0.111 SSH = OK Safe artifacts synced = Start Here and workstation dashboard readback matched; current SHA-256 values are tracked in the workstation dashboard artifact and local sha256sum readback Start Here readback = registry_ready 3, registry_blocked 8, latest_dev_on_gitea 3, production_on_gitea 8, raw_history_sync False Workstation dashboard readback = artifact_sync_synced 2, artifact_sync_blocked 0, MOMO current main baseline ready 2 MOMO Pro Mac Mini workspace = /Users/ogt/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty 0 MOMO Pro MacBook workspace = /Users/ooo/codex-workspaces/momo-pro-dev, branch codex/momo-current-main-dev-base-20260624, commit 84035906aba0e5e190d031a13cfd9b47a8cd1f73, SYSTEM_VERSION V10.653, dirty 0 AwoooGo MacBook workspace = ready on dev commit 8471b376d97c1436d4612ece17f51ba0950f114d, dirty 0 ``` 允許宣告： ```text Mac Mini / MacBook Pro 已同步 Codex 開工入口與治理 snapshot。 MOMO Pro 可以在 Mac Mini / MacBook Pro 從 Gitea current-main Codex baseline 開工；實作前仍需從 codex/momo-current-main-dev-base-20260624 切新的 codex/。 MOMO import-boundary fix 已經由 main / CD #904 部署到 production；後續仍要等下一個真實匯入檔驗證 failure boundary 是否阻止搬檔。 ``` 禁止宣告： ```text raw Codex / ChatGPT 歷史聊天已同步。 所有產品都能雙機同步開發。 把 MOMO Pro 程式版本 V10.653 當成 MOMO 業務資料已更新。 2026FIFA / Agent Bounty owner preflight 已通過。 ``` --- ## 1. Golden Startup Order ```text 0. Freeze automation and preserve evidence 1. Physical/network layer 2. 188 data layer 3. 110 registry/observability layer 4. 120/121 K3s layer 5. AWOOOI workload layer 6. Public routes and alert chain 7. High-load batch/consumer/crawler services 8. Runner/CD 9. AI auto-remediation 10. 112 Kali scanner, if needed ``` Never start runner/CD before 188 PostgreSQL, 110 Harbor, K3s nodes, and AWOOOI API are healthy. ### 1.1 Dependency Graph ```mermaid flowchart TD network["P0 network: LAN, ARP, SSH"] --> data188["188 data: PostgreSQL, Redis, momo DB, SignOz"] network --> obs110["110 registry/observability: Harbor, Gitea, Prometheus, Alertmanager, Sentry"] data188 --> k3s["120/121 K3s: server, agent, VIP, NodePorts"] obs110 --> k3s k3s --> workload["AWOOOI workload: API, Web, K8s Secrets"] workload --> alertchain["Alert chain: Alertmanager webhook, Telegram"] workload --> public["Public routes: awoooi.wooo.work, mo.wooo.work"] public --> schedules["Schedules: cron, CronJobs, backups, exporters"] schedules --> highload["High-load release: crawlers, Snuba, ClickHouse merges, runners/CD"] highload --> ai["AI auto-remediation: limited execution"] ``` This is also captured in the machine-readable baseline: ```text ops/reboot-recovery/full-stack-cold-start-baseline.yml ``` The YAML baseline is the source of truth for: - hosts, roles, and SSH users - phase ordering - service startup dependencies - endpoint success codes - schedule freshness thresholds - stateful-service protection boundaries - AI automation release gates ### 1.2 Phase Gate Logic Each phase has the same decision rule: | Result | Meaning | Action | |--------|---------|--------| | `BLOCKED` | A dependency required by later phases is down. | Stop phase release and fix the first blocked gate. | | `WARN` | Core dependency passed, but confidence is incomplete. | Continue diagnosis, but do not release runner/CD/AI full execution. | | `GREEN` | All checks in scope passed. | Release the next phase only. | The cold-start flow is intentionally conservative: ```text P0 network green -> P0 188 data green -> P0 110 registry/observability green -> P1 K3s green -> P2 workload + alert chain green -> P2 public routes green -> P2 schedules green -> P3 high-load services and runners/CD -> AI auto-remediation limited execution ``` The final release condition is not "containers are running". It is: ```text PASS > 0 WARN = 0 BLOCKED = 0 Result: GREEN ``` ### 1.3 重啟 GO / NO-GO 決策樹 每次維護前先用這張表決定是否可以重啟，以及重啟後可以宣稱到哪個層級。 | 情境 | GO / NO-GO | 可做範圍 | 完成宣告上限 | |------|------------|----------|--------------| | 03:00 offsite sync 正在跑 | `NO-GO` | 只讀觀察，等待 sync 結束後 verifier | 不宣告維護完成 | | 120 不可達，但只重啟 110 | `CONDITIONAL GO` | 只可宣稱 110 / public service recovery；不可跑 120 backup fix | `SERVICE_READY`，不可 `FULL_STACK_GREEN` | | 188 data layer 不健康 | `NO-GO` | 先修 PostgreSQL / Redis / Docker / SignOz / momo DB | 不釋出 K3s / runner / AI | | 110 Harbor / Registry 不健康 | `NO-GO for K3s deploy` | 先修 registry；K3s 可能 image pull 失敗 | 不釋出 CD / deploy | | 120 / 121 都 Ready，offsite verifier 綠 | `GO` | 可做完整 cold-start release chain | 需 scorecard `WARN=0 BLOCKED=0` | | credential escrow marker 缺失 | `GO for service reboot`，`NO-GO for DR complete` | 可恢復服務；不可宣稱 DR scorecard complete | `SERVICE_READY` 或 `BLOCKED_DR_ESCROW` | | Alertmanager required rules 不可見 | `NO-GO for unattended window` | 先修 alert rules / drift guard | 不釋出 AI auto-remediation | GO 只代表允許執行指定範圍，不代表完成。完成一定要回到 §15 Done Criteria。 ### 1.4 Plan B：降級運轉與回復路徑 Plan B 不是另一套可以繞過 preflight 的重啟流程，也不是事故中臨場改主機的授權。Plan B 是當 Plan A 無法在維護窗口內達成 `FULL_STACK_GREEN` 時，預先定義「最低可接受服務目標、停止線、降級等級、主機路徑、回到 Plan A 的條件」。 Plan A 的目標是： ```text B4_FULL_STACK_GREEN：cold-start scorecard WARN=0 / BLOCKED=0，backup、offsite、DB、alert、scheduler、K3s、public route 與業務資料新鮮度都綠。 ``` Plan B 的目標是： ```text 先保住核心服務與資料完整性，不擴大 blast radius，不把部分可用誤報成 full-stack green，並把下一個 blocker 留成可追蹤工單。 ``` Plan B 的機讀契約固定在 `ops/reboot-recovery/full-stack-cold-start-baseline.yml` 的 `plan_b` 區塊；`scripts/reboot-recovery/reboot-recovery-readiness-audit.sh` 必須檢查 SOP 與 baseline 都保留 B0-B5、T+120 停止線與三個收尾狀態。若這些欄位缺失，readiness audit 必須回 `BLOCKED`。 #### Plan B 紅線 | 紅線 | 具體要求 | |------|----------| | 不假綠 | 不用 route 200、pod up、container up、UI 可見、CD success 或單一 smoke pass 宣稱完整恢復。 | | 不消音正確紅燈 | 120 / backup / credential escrow / alert / scheduler 的紅燈如果反映真實缺口，必須保留。 | | 不做未授權寫操作 | 沒有維護窗口與人工批准時，不重啟 Docker daemon、不 reload Nginx、不改 firewall / iptables、不 `kubectl patch` live、不讀 secret、不做 destructive recovery。 | | 不釋出高風險自動化 | CD runner、AI auto-remediation、heavy crawler、batch import、repair bot 必須等依賴鏈綠燈後才解除 freeze。 | #### Plan B 觸發條件 | 觸發條件 | 立即動作 | 可宣稱上限 | |----------|----------|------------| | 03:00 offsite sync、02:00 backup 或 full verifier 仍在跑 | 延後重啟；只讀等待完成 | `B0_ABORTED_BEFORE_REBOOT` | | 任一 P0 主機重啟後 15 分鐘仍 ping / SSH 不可達 | 停止釋出下一層，啟動對應主機路徑 | `B1_HOST_RECOVERY_ONLY` | | 188 PostgreSQL / Redis / momo / SignOz 任一核心資料面不健康 | 凍結 K3s deploy、runner、AI auto-remediation | `B1_HOST_RECOVERY_ONLY` | | 110 Harbor / Gitea / Alertmanager / Prometheus 不健康 | 凍結 CD / deploy / image pull 相關流程 | `B2_CORE_SERVICE_READY` 以下 | | 120 或 121 單台不健康，但另一台 control-plane 可承載 | 進入單節點 K3s 服務模式，保留 HA 紅燈 | `B2_CORE_SERVICE_READY` | | public route 可用，但 DB / backup / alert / schedule 任一不綠 | 標記 `ROUTE_GREEN_ONLY`，不宣稱 service green | `B2_CORE_SERVICE_READY` | | cold-start `WARN>0`、`BLOCKED=0` | 可宣稱服務可用但仍 degraded | `B3_SERVICE_AVAILABLE_DEGRADED` | | credential escrow missing | 可完成服務恢復，不可宣稱 DR complete | `B4_FULL_STACK_GREEN` 或以下，禁止 `B5_DR_COMPLETE` | #### Plan B 主機路徑 | 故障域 | 降級路徑 | 回到 Plan A 的條件 | |--------|----------|--------------------| | 110 失敗 | 保留 120 / 121 K3s 與 188 data；凍結 CD、runner、Harbor image push、Alertmanager outbound；先確認 Gitea / Harbor / Prometheus / Alertmanager 是否只是 host service 層問題。 | 110 `HOST_READY`、Harbor / Gitea / Prometheus / Alertmanager 健康、backup-status 無 110 core blocker、cold-start 110 checks 綠。 | | 120 失敗 | 121 承載 K3s control-plane；保留 `120_DEGRADED` 紅燈；不宣稱 K3s AA；不跑 120 backup fix；必要時走 console / fsck recovery。 | 120 ping / SSH OK、root filesystem rw、`k3s active`、node `mon Ready`、backup-configs / backup-all / offsite / cold-start chain 全過。 | | 121 失敗 | 120 承載 K3s control-plane；保留 `121_DEGRADED` 紅燈；不宣稱 workload balanced；避免非必要 rollout。 | 121 ping / SSH OK、`k3s active`、node `mon1 Ready`、API/Web placement 回到 max skew <= 1。 | | 188 失敗 | 先保資料面：PostgreSQL、Redis、momo DB、SignOz、Ollama / AI provider；凍結會寫入資料或產生大量負載的 batch / crawler / AI flow。 | 188 `HOST_READY`、PostgreSQL / Redis / momo parity / SignOz / AI provider route 健康，且 backup/status 無 188 core blocker。 | | K3s degraded | 保留現有健康 Pod；先查 nodes / pods / events / VIP / NodePort；避免盲目重啟 k3s 或刪 Pod。 | `mon` / `mon1` Ready、API/Web/Worker rollout healthy、public API/Web / alert webhook / scorecard 通過。 | | Public gateway degraded | 保住內部 API / VIP / data；不 reload Nginx、不改 DNS/TLS/certbot/firewall，除非有 owner-approved maintenance window。 | Nginx config owner evidence、route smoke、TLS / ACME、rollback owner 與 post-check 計畫通過。 | #### Plan B 服務等級 維護期間所有回報都必須使用以下等級之一，禁止用「差不多好了」或「應該正常」： | 等級 | 意義 | 最低證據 | |------|------|----------| | `B0_ABORTED_BEFORE_REBOOT` | preflight 發現 NO-GO，取消或延後重啟 | 未做 runtime 寫操作；記錄 NO-GO blocker。 | | `B1_HOST_RECOVERY_ONLY` | 只完成主機層恢復 | 目標主機 ping / SSH / boot time / systemd 基礎狀態可判定；服務尚未全驗。 | | `B2_CORE_SERVICE_READY` | 核心服務可用，但完整依賴鏈未過 | public route、API、DB 或 K3s 主要面可用；backup / alert / scheduler / scorecard 尚未全綠。 | | `B3_SERVICE_AVAILABLE_DEGRADED` | 核心服務可用，cold-start 無 hard block 但仍有 WARN | cold-start `BLOCKED=0`；WARN 被明確列出且不被消音。 | | `B4_FULL_STACK_GREEN` | 本次重啟恢復完成 | cold-start `PASS>0 WARN=0 BLOCKED=0`，backup / offsite / DB / alert / scheduler / data freshness 全綠。 | | `B5_DR_COMPLETE` | DR 完整 | `B4` 加上 credential escrow missing `0`，restore / escrow / offsite evidence 完整。 | #### Plan B 執行時序 ```text T+0 freeze CD / runner / AI auto-remediation / heavy batch；保留 console、journal、backup、scorecard evidence。 T+5 判定 HOST_POWERED / HOST_BOOTED / HOST_READY；任一 P0 host 不可達即進入主機 Plan B。 T+15 188 data 或 110 registry / observability 不健康時停止釋出 K3s、runner、AI。 T+30 public route 可用但 DB / backup / alert / scheduler 未過時，只能回報 B2，不得 full green。 T+60 必須跑 cold-start scorecard；若仍 WARN / BLOCKED，留下 Plan B 等級與下一個 blocker。 T+120 若仍未達 B4，開 incident / follow-up，不延長窗口做未授權 runtime 寫操作。 ``` #### Plan B 收尾條件 Plan B 只能以下列三種狀態收尾： | 收尾狀態 | 條件 | 下一步 | |----------|------|--------| | `RETURNED_TO_PLAN_A` | blocker 已清，完成 Plan A 全鏈路驗證 | 更新 reboot ledger，記錄實際耗時與 SOP 差異。 | | `SERVICE_AVAILABLE_DEGRADED` | 服務可用但 scorecard 仍 WARN，或 DR / escrow / governance gate 未完成 | 保留紅燈，開下一步 owner / evidence / maintenance task。 | | `OPEN_INCIDENT_REQUIRED` | P0 host、data、K3s、gateway、backup、alert 任一仍 hard blocked | 停止維護窗口，保留 evidence，升級事故處理。 | Plan B 的專業標準不是「保證每次都綠」，而是保證每次重啟都能快速知道現在到哪一層、什麼不能宣稱、下一個 blocker 是誰、以及是否可以安全回到 Plan A。 --- ## 2. Automation Freeze Cold start creates noisy metrics and partial failures. During P0/P1, keep automation in observe-only mode. | Item | Cold-start policy | Reason | |------|-------------------|--------| | Gitea/GitHub runners | Last | Build jobs can saturate 110 CPU/RAM. | | momo-scheduler / crawlers | Last | Chrome and batch work can saturate 188. | | Sentry/Snuba consumers | Controlled | Kafka backlog and ClickHouse merge can create temporary high load. | | Alertmanager outbound notification | Gate | Avoid alert storms before API webhook and Telegram are verified. | | AI auto-repair | Observe-only | Metrics, Redis, KM, and playbooks may be incomplete. | | Stateful DB restart | Human approval | PostgreSQL, Redis, ClickHouse, Harbor DB, Sentry DB are not generic restart targets. | ### 2.1 Freeze 執行清單 進入維護窗口後，先把「會放大事故」的來源降到 observe-only 或延後釋出。若沒有做到這一步，後續負載和告警會很難判讀。 | 順序 | 對象 | 只讀確認 | 允許動作 | 禁止動作 | |------|------|----------|----------|----------| | 1 | runner / CD | `systemctl list-units "actions.runner.*"`、Gitea Actions running jobs | 暫停新 job、等待可完成 job 結束 | 重啟 Docker daemon 來中斷 job | | 2 | AI auto-remediation | Prometheus / Alertmanager / cold-start monitor 狀態 | 切 observe-only、保留告警 | 自動 restart stateful service | | 3 | momo scheduler / crawler | container health、recent logs、DB parity | 延後 heavy import、保留現有資料 | 在 DB 未綠時強行匯入 | | 4 | Sentry / Snuba | ClickHouse / Kafka health、consumer restart loop | 控制 consumer 釋出順序 | generic compose down/up 全套重啟 | | 5 | K3s workload | node readiness、pods、events | 依 node 狀態 cordon/drain | 120 不可達時宣稱 drain 成功 | 多個工作視窗同時處理事故時，第一優先是避免互相打斷：只要有人在收斂 Docker / Nginx / firewall / K3s 寫操作，其他視窗先只讀觀察，直到明確交接。 ### 2.2 CD / SSH Trust Guardrail 2026-06-13 的冷啟動假紅燈顯示：CD workflow 若用 `ssh-keyscan ... > /home/wooo/.ssh/known_hosts`，會覆蓋 110 使用者層的全域 SSH trust，導致 110 到 120 / 188 的 strict SSH 檢查失敗。這會把實際已恢復的主機誤判成 blocked。 固定規則： | 項目 | 正確做法 | 禁止 | |------|----------|------| | Deploy 專用 host key | 寫入 `/home/wooo/.ssh/deploy_known_hosts` | 寫入或覆蓋 `/home/wooo/.ssh/known_hosts` | | Deploy SSH options | `-o UserKnownHostsFile=/home/wooo/.ssh/deploy_known_hosts` | 共用 operator / cold-start 的 `known_hosts` | | 冷啟動 SSH trust | 保留 120 / 188 的已驗證 fingerprint；修復前先備份 | 無 fingerprint 交叉驗證就 `ssh-keygen -R` 或重建全檔 | | 驗證 | CD 後檢查 `known_hosts` mtime、120 / 188 entries、strict SSH | 只看 CD success badge | 2026-06-13 修復錨點： - Source fix：Gitea main 包含 `80e6ec1a fix(ci): avoid clobbering runner known hosts`。 - Deploy marker：`e4a349bc chore(cd): deploy 414413a [skip ci]` 後，`/home/wooo/.ssh/known_hosts` mtime 仍停在 `2026-06-13 01:20:02 +0800`，未被 CD 覆蓋。 - Deploy isolated file：`/home/wooo/.ssh/deploy_known_hosts` mtime `2026-06-13 01:24:05 +0800`。 - Global strict entries：120 ED25519 line 4、188 ED25519 line 5 仍存在；strict SSH 到 `wooo@192.168.0.120` 與 `ollama@192.168.0.188` 必須通過。 --- ## 3. P0 Evidence And Network Run from any machine on the same LAN: ```bash for h in 110 120 121 188; do ping -c 2 -W 2 192.168.0.$h >/dev/null && echo "PING_OK 192.168.0.$h" || echo "PING_FAIL 192.168.0.$h" done arp -an | grep -E '192\.168\.0\.(110|120|121|188)' for h in 110 120 121 188; do nc -G 3 -z 192.168.0.$h 22 && echo "SSH_OK 192.168.0.$h" || echo "SSH_FAIL 192.168.0.$h" done ``` Then capture reboot evidence: ```bash ssh ollama@192.168.0.188 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20' ssh wooo@192.168.0.110 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20' ssh wooo@192.168.0.120 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20' ssh wooo@192.168.0.121 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20' ``` If any host has ARP `incomplete` or SSH port down, stop here and fix physical/network first. ### 3.1 主機已啟動判定標準 每台主機重啟後先跑「四段式啟動判定」。只有全部符合角色期望，才進入服務恢復。 ```bash for h in 110 120 121 188; do ip="192.168.0.$h" echo "=== $ip ===" ping -c 2 -W 2 "$ip" >/dev/null && echo "HOST_POWERED_OR_LAN_OK=1" || echo "HOST_POWERED_OR_LAN_OK=0" arp -an | grep "$ip" || true nc -G 3 -z "$ip" 22 && echo "SSH_PORT_OPEN=1" || echo "SSH_PORT_OPEN=0" done ``` 可 SSH 後： ```bash ssh wooo@192.168.0.110 'hostname; date; who -b; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true; free -h; swapon --show' ssh wooo@192.168.0.121 'hostname; date; who -b; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true' ssh ollama@192.168.0.188 'hostname; date; who -b; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true; free -h' ``` 120 若不可 SSH，狀態只能是 `HOST_POWERED_UNKNOWN` 或 `HOST_BOOTED_UNKNOWN`，必須走 console / VM / network 檢查，不可用 Kubernetes stale node object 代替主機現況。 | 判定 | 必要條件 | 下一步 | |------|----------|--------| | `HOST_BOOTED` | ping 或 ARP 有回應、SSH port open、`who -b` 顯示本次 boot time | 檢查角色服務 | | `HOST_READY` | `systemctl is-system-running` 為 `running`，或 degraded units 已逐一解釋且不影響本 host 角色 | 進入服務層驗證 | | `HOST_DEGRADED` | failed units 存在且影響本 host 角色，或 swap 滿、root readonly、boot storage error | 先修 host，不釋出下一層 | | `HOST_UNREACHABLE` | ping/SSH/ARP 失敗 | 停止遠端修復假設，改 console/VM/network | 2026-06-12 110 事故教訓：failed unit 若指向不存在的 legacy 路徑，要先確認是否仍屬現役服務。停用 stale timer 可以解除 host degraded，但必須同步 source-of-truth 後續清理，不能靠反覆 `reset-failed` 掩蓋。 2026-06-26 188 事故教訓：PostgreSQL host cluster / Docker product DB / exporter 三者必須分開判定。`pg_isready`、`pg_up=1` 或 public route `200` 只能證明某個 PostgreSQL endpoint 可用，不能證明 `postgresql@14-main` 已恢復。若 journal 出現 `could not locate a valid checkpoint record`，不得由 startup 腳本或 AI 自動執行 `pg_resetwal`；必須進入 DB owner / backup restore / maintenance window / rollback owner / post-check gate。 --- ## 4. P0 188 Data Layer 188 is the first real service dependency because multiple product data planes, exporters, and AI / observability services depend on PostgreSQL-compatible endpoints. Do not assume the host cluster `postgresql@14-main`, Docker product databases, and exporter target are the same endpoint; prove the authoritative endpoint before repair. ### 4.1 Startup order 1. `containerd` 2. `docker` 3. `postgresql@14-main` 4. `k3s_datastore.kine` maintenance 5. `redis-server` on `6380` 6. `ollama` or current AI proxy dependencies 7. `nginx` 8. Docker networks 9. MinIO / OpenClaw / SignOz 10. momo / litellm / batch services after load is stable ### 4.2 Read-only check ```bash ssh ollama@192.168.0.188 ' hostname; date; uptime; free -h systemctl is-active containerd docker postgresql@14-main redis-server ollama nginx || true pg_lsclusters 2>/dev/null || true ss -ltnp "sport = :5432" 2>/dev/null || ss -ltn "sport = :5432" || true pg_isready -h localhost -p 5432 || true redis-cli -p 6380 ping 2>/dev/null || redis-cli ping 2>/dev/null || true docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | head -120 ' ``` ### 4.3 PostgreSQL WAL checkpoint damage Signature: ```text PANIC: could not locate a valid checkpoint record invalid primary checkpoint record unexpected pageaddr ... in log segment ... ``` This may block if the affected cluster is the authoritative runtime datastore: - `188:5432` - K3s startup on 120/121 - AWOOOI API DB access - Alertmanager webhook if API cannot start 2026-06-26 counterexample: host cluster `14/main` can be down while product DB containers and exporters still serve traffic. Therefore `pg_isready` is not enough and failed `postgresql@14-main` is not automatically a product outage. First map the listening process / container, current app DB configuration, and backup freshness. Break-glass example only after DB owner approval, backup evidence, maintenance window, rollback owner, and post-check plan: ```bash sudo systemctl stop postgresql@14-main sudo install -d -m 700 -o postgres -g postgres /var/backups/postgresql sudo tar -C /var/lib/postgresql/14 -czf /var/backups/postgresql/14-main-before-pg-resetwal-$(date +%Y%m%d-%H%M%S).tgz main sudo -u postgres /usr/lib/postgresql/14/bin/pg_resetwal -f /var/lib/postgresql/14/main sudo systemctl start postgresql@14-main pg_isready -h localhost -p 5432 sudo -u postgres psql -d k3s_datastore -c "VACUUM ANALYZE kine;" ``` Do not run `pg_resetwal`, `DROP`, reinitialize the cluster, delete `/var/lib/postgresql`, or restore an old backup from AI/startup automation. These are break-glass actions only. --- ## 5. P0/P1 110 Registry And Observability 110 must recover Harbor/Gitea/Monitoring early, but runners last. ### 5.1 Startup order 1. `docker` 2. Remove `Exited (128)` / `Exited (137)` orphan containers 3. Harbor `harbor-log` 4. Harbor full stack 5. Gitea 6. Prometheus / Alertmanager / Grafana / exporters 7. Langfuse 8. SignOz 9. Sentry DB layer 10. Sentry web/worker/consumer layer 11. Gitea host runner and actions runners ### 5.2 Checks ```bash ssh wooo@192.168.0.110 ' hostname; date; uptime; free -h systemctl is-active docker || true curl -s -o /dev/null -w "harbor=%{http_code}\n" --max-time 5 http://127.0.0.1:5000/v2/ || true curl -s -o /dev/null -w "gitea=%{http_code}\n" --max-time 5 http://127.0.0.1:3001/ || true curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true curl -s -o /dev/null -w "sentry=%{http_code}\n" --max-time 10 http://127.0.0.1:9000/ || true docker ps --format "{{.Names}}\t{{.Status}}" | head -120 ' ``` Harbor healthy means `/v2/` returns `200` or `401`. Do not treat `401` as failure. ### 5.3 Runner gate Runner may start only after all are true: - `188 PostgreSQL` ready - `110 Harbor` ready - `110 Gitea` ready - `120/121 K3s` nodes ready - AWOOOI API health passes - 110 load/core is below `1.0` for at least 15 minutes - runner systemd guardrails are active: `CPUQuota=200%`, `MemoryMax=2G`, `WatchdogUSec=0` Check: ```bash ssh wooo@192.168.0.110 ' for u in $(systemctl list-units "actions.runner.*" --all --no-legend --plain | awk "{print \$1}"); do echo "=== $u ===" systemctl show "$u" -p ActiveState -p SubState -p CPUQuotaPerSecUSec -p MemoryMax -p WatchdogUSec -p NRestarts done ' ``` If `WatchdogUSec` is not `0`, apply the guardrail script manually with sudo: ```bash sudo /home/wooo/scripts/apply-runner-systemd-guardrails.sh --apply ``` --- ## 6. P1 120/121 K3s K3s must wait for 188 PostgreSQL and 110 Harbor. ### 6.1 Startup order 1. 120 `k3s.service` 2. 121 `k3s.service`, `k3s-agent.service`, or its live role 3. CNI / kube-proxy 4. Nodes Ready 5. Core pods 6. `awoooi-prod` pods 7. keepalived VIP `192.168.0.125` 8. NodePorts `32334` and `32335` ### 6.2 Checks ```bash ssh wooo@192.168.0.120 ' hostname; uptime pg_isready -h 192.168.0.188 -p 5432 || true systemctl is-active k3s k3s-agent keepalived 2>/dev/null || true kubectl get nodes -o wide 2>/dev/null || true kubectl get pods -A 2>/dev/null | grep -v -E "Running|Completed" || true kubectl get pods -n awoooi-prod -o wide 2>/dev/null || true ip addr show | grep 192.168.0.125 || true ' ssh wooo@192.168.0.121 ' hostname; uptime systemctl is-active k3s k3s-agent keepalived 2>/dev/null || true ip addr show | grep 192.168.0.125 || true ' ``` If K3s is `activating` while 188 PostgreSQL is down, fix PostgreSQL first. Restarting K3s repeatedly will not solve it. ### 6.3 120 / 121 AA / AS 與負載平衡判定 2026-06-12 15:19 live check 確認 120 / 121 都是 K3s `control-plane`，且兩台都是 `k3s active`、`k3s-agent inactive`。因此它們是 K3s 控制面 AA，不是傳統一主一從 AS。 但控制面 AA 不等於業務 workload AA。120 剛從 root filesystem fsck 恢復後，大多數 ArgoCD / AWOOOI / Velero / kube-system workload 仍集中在 121；120 主要只有 DaemonSet 類 Pod。每次 120 / 121 重啟或恢復後，都要額外跑 Pod 落點檢查： ```bash ssh wooo@192.168.0.120 ' sudo kubectl get nodes -o wide sudo kubectl get pods -A -o wide sudo kubectl top nodes 2>/dev/null || true sudo kubectl top pods -A --sort-by=cpu 2>/dev/null | head -30 || true ' ``` 判定規則： | 判定 | 條件 | 可宣稱 | |------|------|--------| | `K3S_CONTROL_PLANE_AA` | 120 / 121 都是 `Ready control-plane` | 控制面雙節點可用 | | `WORKLOAD_IMBALANCED` | 主要 deployment / pod 都落在單一節點 | 不可宣稱服務 AA；需排程治理 | | `WORKLOAD_BALANCED` | replicas >= 2 的核心 API / Web 跨 120 / 121 分散 | 可宣稱承載層分散 | | `STATEFUL_AA` | storage replication、backup / restore drill、failover drill 都通過 | 才可宣稱資料層 AA | 負載平衡與遷移評估的正式基準文件是 `docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md`。恢復期先完成 P0 備份鏈與 cold-start scorecard，再做 topology spread 或服務搬遷。 --- ## 7. P2 AWOOOI Workloads Run after K3s nodes are Ready: ```bash ssh wooo@192.168.0.120 ' kubectl get deploy -n awoooi-prod kubectl get pods -n awoooi-prod -o wide kubectl get svc -n awoooi-prod kubectl get events -n awoooi-prod --sort-by=.lastTimestamp | tail -40 ' curl -s --max-time 8 http://192.168.0.125:32334/api/v1/health curl -s -o /dev/null -w "web=%{http_code}\n" --max-time 8 http://192.168.0.125:32335/ ``` If pods are `ImagePullBackOff`, go back to 110 Harbor. If API health fails because DB/Redis is down, go back to 188. --- ## 8. P2 Alert Chain Current main path: ```text Prometheus/Alertmanager on 110 -> http://192.168.0.125:32334/api/v1/webhooks/alertmanager -> AWOOOI API -> TelegramGateway -> Telegram ``` Alertmanager health alone is not enough. The test must be run from 110 or another host that can reach the K3s VIP / NodePort. A local workstation connection failure to `192.168.0.125:32334` does not by itself prove Alertmanager cannot deliver alerts, because Alertmanager also runs on 110. Run E2E: ```bash curl -s -X POST http://192.168.0.125:32334/api/v1/webhooks/alertmanager \ -H 'Content-Type: application/json' \ -d '{"receiver":"cold-start-test","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"ColdStartE2ETest","severity":"info"},"annotations":{"summary":"Cold start E2E test, ignore"},"startsAt":"2026-05-05T11:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":""}],"groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"cold-start-test"}' ``` Expected: API returns success and Telegram receives the test alert. 2026-07-01 live readback：from 110, `/api/v1/webhooks/health` returns `200`, `/api/v1/telegram/health` returns `configured`, and synthetic no-secret `CodexAlertChainSmoke` POSTs to `192.168.0.125` / `192.168.0.120` / `192.168.0.121` all returned HTTP `200` with `告警已排入背景分析`. If Telegram is still not visible to the owner, the next check is AWOOOI API background analysis / TelegramGateway outbound status, not Prometheus alert existence. --- ## 9. P2 Schedules And Delayed Work Do not mark the reboot complete until scheduled work is proven runnable. A container can be healthy while its cron path is broken. | Host / Layer | Required check | Success baseline | |--------------|----------------|------------------| | 188 cron | `systemctl is-active cron` and `crontab -l` | cron active; backup, restart exporter, stats exporter entries present | | 188 backup-from-110 | `backup_110_last_success_timestamp` in textfile/Prometheus | last success age `< 25h` | | 188 momo-scheduler | `docker inspect momo-scheduler` and `docker logs --since 6h momo-scheduler` | container `running healthy`; `全部排程任務已註冊`; Google Drive auth works; dashboard URLs use container-reachable hostnames | | 188 momo import | manual `run_auto_import_task()` after parser changes | selected sheet is `即時業績明細`; imported date range has matching rows in `daily_sales_snapshot` and `realtime_sales_monthly` | | 110 cron | `systemctl is-active cron` | cron active; Docker/systemd textfile exporters fresh | | 110 startup units | `systemctl --failed` | zero failed units; stale `momo-startup-complete` and `wooo-staggered-startup` disabled | | 120 K8s CronJobs | `kubectl get cronjobs -n awoooi-prod` | unsuspended; no failed Jobs remain after current validation | | 121 DR drill | `crontab -l` | DR drill cron present unless explicitly paused | Useful checks: ```bash ssh ollama@192.168.0.188 'systemctl is-active cron; crontab -l; ls -l /home/ollama/node_exporter_textfiles/*.prom' ssh wooo@192.168.0.110 'systemctl --failed --no-pager; systemctl is-active cron; crontab -l' ssh wooo@192.168.0.120 'sudo kubectl get cronjobs,jobs -n awoooi-prod' ssh wooo@192.168.0.121 'systemctl is-active cron; crontab -l' ``` If a schedule succeeds but emits a false verification alert, fix the verification rule before releasing AI auto-remediation. False positives train operators to ignore real alarms. --- ## 10. P2/P3 Stateful Service Guardrails | Tier | Examples | Automation | |------|----------|------------| | BLOCK | PostgreSQL data dir, ClickHouse data dir, Harbor DB, Sentry DB | No automatic destructive action. Human approval only. | | CRITICAL_HITL | Redis, Kafka, MinIO, SignOz ClickHouse, Sentry ClickHouse | Human-in-the-loop restart/repair. | | STANDARD_HITL | API/Web/worker, OpenClaw, litellm | Restart only with evidence and blast-radius check. | | AUTO | Stateless exporters, blackbox, nginx exporter | Auto restart allowed after verification. | Never use generic `docker restart $(docker ps -q)` during cold start. ### 10.1 Dirty-Reboot Storage Corruption Treat these log signatures as storage corruption, not ordinary service flakiness: - `Bad message` - `Structure needs cleaning` - `Unknown codec` - `PANIC: could not locate a valid checkpoint record` - Kafka `Malformed line` in checkpoint files - ClickHouse `broken and needs manual correction` Cold-start automation may stop a restart storm and collect evidence, but it must not delete the original data directory. If a filesystem returns `Bad message` or `Structure needs cleaning`, the real root cause is below the container layer. Online recovery can restore service from readable data, but complete historical recovery requires an offline filesystem check or backup restore. ### 10.2 ClickHouse Clean-Clone Recovery Pattern Use this pattern for Sentry ClickHouse or SignOz ClickHouse when individual corrupted parts cannot be moved because the host filesystem rejects reads. ```text 1. Stop the compose stack or at least stop dependent consumers. 2. Disable restart loops for the failing container. 3. Save logs and build an exclude list from unreadable store paths. 4. Preserve the original volume as _data.corrupt-YYYYMMDD-HHMMSS. 5. Create a clean _data clone with readable files only. 6. Add flags/force_restore_data. 7. Start ClickHouse first, then web/API, then consumers. 8. Verify HTTP, merge backlog, and restart count before releasing high-load services. ``` Do not replace this with `rm -rf store/...` unless the unreadable path is already backed up or the commander explicitly accepts data loss. The preferred incident artifact is: ```text /var/lib/docker/volumes//_data.corrupt-YYYYMMDD-HHMMSS /var/backups/--YYYYMMDD-HHMMSS ``` ### 10.3 Kafka Checkpoint Recovery Pattern If Kafka refuses to start with malformed checkpoint files after a dirty reboot, preserve and move only checkpoint files: ```text log-start-offset-checkpoint recovery-point-offset-checkpoint replication-offset-checkpoint ``` Then start Kafka and confirm health before starting Snuba/Sentry consumers. Do not delete topic directories or Kafka logs during cold-start recovery. --- ## 11. P3 High-Load Services Only release these after P0/P1/P2 gates are green: | Host | Service | Release condition | |------|---------|-------------------| | 188 | momo-scheduler / crawler | load/core < 1.0 for 15 minutes and DB healthy | | 188 | SignOz ClickHouse | healthy and merge backlog trending down | | 188 | litellm | `/health/liveliness` good and provider route verified | | 110 | Sentry Snuba consumers | ClickHouse healthy and Kafka backlog decreasing | | 110 | Sentry uptime-checker | Sentry web/DB healthy | | 110 | runners | all previous gates green, `host_runaway_process.prom` fresh, orphan browser group count `0`, and load/core < 1.0 for 15 minutes unless the remaining load is explicitly attributed to active CI | ### 11.1 110 Runaway Browser / CI Load 分流 2026-06-18 110 CPU 滿載事件證明：泛用 `HostHighCpuLoad` 只能說主機忙，不能告訴 operator 要不要殺程序。110 現在必須使用專用 host runaway process 指標做第一層分流： ```bash grep -E 'awoooi_host_runaway_|awoooi_host_gitea_actions_|awoooi_host_load5_per_core|awoooi_host_swap_used_ratio' \ /home/wooo/node_exporter_textfiles/host_runaway_process.prom ``` Prometheus 也必須讀得到同一份 textfile；2026-06-18 14:31-14:32 live scrape 已確認 `awoooi_host_runaway_process_monitor_up{host="110"}=1`、orphan group count `0`、active CI container count `2`、`remediation_authorized=0`，且 missing / orphan alerts 均未 firing。 判讀： | 指標組合 | 判定 | 行動 | |----------|------|------| | `awoooi_host_runaway_browser_orphan_group_count > 0` 且 CPU `>= 100` | orphan headless browser / smoke process group | 執行 `host-runaway-process-remediation.py` dry-run；人工確認後才可 gated `SIGTERM` | | orphan count `0` 且 `awoooi_host_gitea_actions_active_container_count > 0` | 合法 CI build/test 負載 | 觀察 Gitea Actions queue / workflow timeout；不殺程序 | | `awoooi_host_runaway_process_monitor_up` 缺失或 stale | 監控盲區 | 修 cron / textfile collector / Ansible role，不宣稱 AI Ops 可觀測 | | `awoooi_host_runaway_process_remediation_authorized > 0` | 監控器被誤改成執行器 | 立即回滾；runtime remediation 必須只走 gated helper | 正式 PlayBook： ```text docs/runbooks/HOST-RUNAWAY-PROCESS-AIOPS-PLAYBOOK.md ``` 這條 PlayBook 不取代 Docker / Sentry / Harbor / K3s / backup SOP。它只處理 orphan browser smoke 與 CI load 分類，避免 CPU 高時誤重啟 Docker 或誤殺合法 build。 --- ## 12. Baseline And AI Auto-Remediation Gate ### 12.1 Stable Runtime Baseline These are release gates after the first cold-start recovery pass: | Area | Baseline | |------|----------| | 188 host | PostgreSQL accepting, Redis PONG, momo `/health` 200, SignOz HTTP reachable, load/core < 1.0 sustained before crawlers | | 110 host | Harbor `/v2/` 200/401, Gitea 200/302, Prometheus ready, Alertmanager healthy, Sentry HTTP 200/302/400, no ClickHouse/Kafka restart loop | | K3s | 120/121 nodes Ready, VIP `192.168.0.125` present, AWOOOI API 2xx/3xx, Web 2xx/3xx | | Public routes | `https://awoooi.wooo.work/api/v1/health` 2xx/3xx, `https://mo.wooo.work/health` 2xx/3xx | | Guardrails | Docker/systemd/storage/backup/runaway-process textfile exporters fresh, runner `CPUQuota=200%`, `MemoryMax=2G`, `WatchdogUSec=0` | | Schedules | cron active on 110/188/120/121; K8s CronJobs unsuspended; no current failed Jobs; 188 backup success `< 25h` | | Backlog | ClickHouse merges and Kafka/Snuba lag trending down, not increasing for two consecutive checks | If service health is green but load average remains high, check live CPU and IO before changing memory limits. High load after Sentry/Snuba or ClickHouse startup can be backlog drain; high CPU from runners/builds/crawlers is a release-order problem. ### 12.2 AI Auto-Remediation Gate AI auto-repair can move from observe-only to limited execution only after: - Prometheus rules are loaded. - docker/systemd textfile exporter files are fresh. - runaway process textfile exporter is fresh and `remediation_authorized=0`. - blackbox probes have stable results. - cron/CronJob schedule checks are green. - AWOOOI API `/api/v1/health` passes. - Alertmanager E2E webhook passes. - Redis/KM/playbook health is available. - No active restart storm. - Host load/core remains below `1.0` for 15 minutes. Until then: - diagnose only - notify only - require human approval for remediation - no DB/ClickHouse/Harbor/Sentry destructive action - no generic restart action against stateful services - no process kill unless `host-runaway-process-remediation.py` has dry-run evidence plus owner approval, maintenance window, and evidence ref --- ## 13. One-Command Readiness Script ### 13.1 Single Pass Run this when you want one read-only snapshot: ```bash bash scripts/reboot-recovery/full-stack-cold-start-check.sh ``` The script is read-only. It does not restart services, delete data, change memory/CPU limits, or patch Kubernetes. It reports gates: - `P0-NETWORK` - `P0-188-DATA` - `P0-110-REGISTRY-OBSERVABILITY` - `P1-K3S` - `P2-WORKLOAD-ALERTCHAIN` - `P2-PUBLIC-ROUTES` - `P2-SCHEDULES` - runner guardrail state inside `P0-110-REGISTRY-OBSERVABILITY` If it prints `BLOCKED`, fix the first blocked gate before moving forward. ### 13.2 Professional Watch Mode Run this after a full reboot when you want the machine to keep checking until the whole stack is ready: ```bash bash scripts/reboot-recovery/full-stack-cold-start-check.sh \ --watch \ --interval 60 \ --max-attempts 30 \ --send-alert-test ``` This is the standard next-reboot release command. It checks every 60 seconds for up to 30 attempts and exits only when the stack is `GREEN` or the last attempt remains degraded/blocked. Use `--send-alert-test` for final release because the alert chain is not proven until the AWOOOI Alertmanager webhook accepts a real POST. Without `--send-alert-test`, the script intentionally leaves a warning so operators do not falsely mark alerting as complete. ### 13.3 Persistent Read-Only Monitor After recovery, host 110 should run the same gate as a node-exporter textfile monitor: ```bash bash scripts/reboot-recovery/install-cold-start-monitor-110.sh ``` This command is not read-only. It copies scripts to 110, rewrites the marked `wooo` crontab block, and immediately refreshes the textfile metric. Run it only inside an approved maintenance window or explicit owner-approved live-sync change. This installs two scripts under `/home/wooo/scripts/`, adds a marked user-cron block, and writes: - `/home/wooo/node_exporter_textfiles/cold_start_recovery.prom` - `/home/wooo/reboot-recovery/cold-start-last.log` The cron path uses `--monitor-read-only`, so it does not POST Alertmanager smoke events every 10 minutes. It converts the cold-start gate into Prometheus metrics: - `awoooi_cold_start_monitor_up` - `awoooi_cold_start_pass_gates` - `awoooi_cold_start_warn_gates` - `awoooi_cold_start_blocked_gates` - `awoooi_cold_start_last_run_timestamp` - `awoooi_cold_start_last_green_timestamp` - `awoooi_cold_start_last_result{result="green|degraded|blocked|check_failed"}` Prometheus rules in `ops/monitoring/alerts-unified.yml` alert when the monitor is missing, stale, blocked, degraded, or has not been green for more than 6 hours. #### 13.3.1 110 cold-start monitor live-sync gate Use this gate whenever the repo-side cold-start script changes. This prevents a false-green where repo evidence is newer than the live 110 monitor. Current read-only evidence, 2026-07-01 18:40 Asia/Taipei: ```text Repo script hash: e320c061f5afd31c2a682576218f549b683f25dafd43dd52acc13b6283b33712 110 live script hash: e320c061f5afd31c2a682576218f549b683f25dafd43dd52acc13b6283b33712 Exporter hash: c52ea4fe8dd58688a87c01ca6288f8f6050aeb82417852213db3e2be69b29568 verify result: COLD_START_MONITOR_DEPLOY_PARITY_OK runtime state: monitor_up=1 warn=8 blocked=2 green=0 blocked_state=1 alerts: ColdStartLastGreenTooOld, ColdStartRecoveryBlocked, ColdStartRecoveryDegraded ``` Read-only verification: ```bash bash scripts/reboot-recovery/verify-cold-start-monitor-deploy.sh ``` Approved apply path, only after maintenance-window / owner approval: ```bash bash scripts/reboot-recovery/install-cold-start-monitor-110.sh bash scripts/reboot-recovery/verify-cold-start-monitor-deploy.sh /home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1 ``` Completion criteria: - `verify-cold-start-monitor-deploy.sh` reports hash parity for `full-stack-cold-start-check.sh` and `cold-start-textfile-exporter.sh`. - `verify-cold-start-monitor-deploy.sh` reports host-key policy and monitor-up parity; it must not require runtime green, because SignOz / MOMO / DR blockers belong to cold-start and recovery scorecards. - The live 110 cold-start output includes the expected current fields, including `MOMO_SOURCE_EMPTY_EVIDENCE_LINES`, `MOMO_IMPORT_CONFIG`, and `MOMO_LATEST_IMPORT_JOB` while MOMO data freshness remains blocked by source absence. - The textfile monitor refreshes without creating alert spam. - LOGBOOK records local hash, remote hash, command type, approval reference, and final cold-start result. NO-GO: - Do not run the installer as part of routine read-only triage. - Do not call repo-side v1.42 deployed on 110 while the hash mismatch remains. - Do not patch 110 manually with ad hoc `scp`; use the existing installer or Ansible source-of-truth path under an approved change. - Do not use deploy parity as a proxy for full-stack green; a synced monitor can still correctly report `blocked`. ### 13.4 Script-To-SOP Coverage Map | Script gate | SOP coverage | Blocks | |-------------|--------------|--------| | `P0-NETWORK` | host reachability, ARP, SSH | every later phase | | `P0-188-DATA` | PostgreSQL, Redis, momo, SignOz | K3s, AWOOOI API, momo public site | | `P0-110-REGISTRY-OBSERVABILITY` | Harbor, Gitea, Prometheus, Alertmanager, Sentry, runner quotas | image pulls, CD, alert rules, runners | | `P1-K3S` | 120/121 K3s, VIP, node readiness, pod health | workload and webhook health | | `P2-WORKLOAD-ALERTCHAIN` | AWOOOI API/Web, Alertmanager webhook | AI auto-remediation and alert confidence | | `P2-PUBLIC-ROUTES` | external AWOOOI and momo URLs | external release | | `P2-SCHEDULES` | cron, CronJobs, backups, textfile exporters, DR drill | final done criteria | ### 13.5 Next-Reboot Operator Contract 1. Run the watch command above. 2. If it stops at `BLOCKED`, repair the first blocked gate and rerun watch mode. 3. If it stops at `WARN`, do not release runner/CD/AI full execution; clear or explicitly accept each warning. 4. Release high-load services only after `GREEN` and load/core stays below `1.0` for 15 minutes. 5. Record the final output summary and any manual repair in `docs/LOGBOOK.md`. ### 13.6 2026-05-29 補充：188 Public Gateway 與備份告警 `aiops.wooo.work` 的 188 public gateway 不可再指向單一 `192.168.0.120:31234/31235`。120 失聯時這會讓 public route 直接 502。正式 baseline 必須走 K3s VIP： ```nginx location /api/ { proxy_pass http://192.168.0.125:32334/api/; } location /api/v1/ws { proxy_pass http://192.168.0.125:32334/api/v1/ws; } location / { proxy_pass http://192.168.0.125:32335; } ``` 變更來源必須是 `infra/ansible/roles/nginx/templates/188-all-sites.conf.j2`，再用 `infra/ansible/playbooks/nginx-sync.yml` 收斂；禁止只改 188 live 檔而不回寫 Ansible baseline。 備份告警有兩層，缺一不可： - `ops/monitoring/alerts-unified.yml` 是 repo canonical。 - 110 live `/home/wooo/monitoring/alerts.yml` 與 `/home/wooo/monitoring/alerts-unified.canonical.yml` 必須一致，否則 `prometheus-rule-drift-guard` 可能把規則拉回舊版。 重啟後必查： ```bash curl -s http://127.0.0.1:9090/api/v1/rules \ | python3 -c 'import json,sys; d=json.load(sys.stdin); names=[r.get("name") for g in d["data"]["groups"] for r in g["rules"]]; print([n for n in ["BackupAggregateRunFailed","BackupConfigCapturePartial","BackupOffsiteCopyStale","BackupCredentialEscrowEvidenceMissing","ColdStartRecoveryBlocked"] if n not in names])' cat /home/wooo/node_exporter_textfiles/prometheus_rule_drift_guard.prom ``` 若 120 尚未恢復，`BackupConfigCapturePartial{target="120-k3s-host-configs"}` 與 cold-start blocked 是正確訊號，不可消音。120 恢復後再重跑： ```bash /backup/scripts/backup-configs.sh /backup/scripts/backup-all.sh /backup/scripts/sync-offsite-backups.sh --mode sync /backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color ``` ### 13.7 2026-05-29 補充：momo PostgreSQL Index 與資料同步 `mo.wooo.work` 不能只看 `/health` 或首頁 200。重啟或 fsck 後，PostgreSQL index 可能讓匯入流程表面完成，但 `daily_sales_snapshot` 未同步到 `realtime_sales_monthly`。本次症狀： - `daily_sales_snapshot` 已有 2026-05-01 到 2026-05-28 的 17,353 筆。 - `realtime_sales_monthly` 同日期範圍為 0 筆。 - momo-scheduler log 出現 PostgreSQL 內部錯誤 `posting list tuple ... cannot be split`。 標準處理順序： ```bash # 188 / momo-db，只重建索引，不刪資料 docker exec -i momo-db bash -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1' <<'SQL' REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly; SQL ``` 重建索引後，才可針對缺漏日期做 idempotent 補同步。正式作法必須先確認 `realtime_sales_monthly` 該日期範圍筆數，若非 0，需先保存查詢結果並確認是否重跑同範圍同步；不可整表 truncate、不可整庫 restore。補同步後至少驗證： ```sql SELECT count(*), min(snapshot_date::date), max(snapshot_date::date) FROM daily_sales_snapshot WHERE snapshot_date::date BETWEEN DATE '2026-05-01' AND DATE '2026-05-28'; SELECT count(*), min("日期"::date), max("日期"::date) FROM realtime_sales_monthly WHERE "日期"::date BETWEEN DATE '2026-05-01' AND DATE '2026-05-28'; ``` 兩張表同日期範圍筆數與日期上下界必須一致。完成後清除 momo 應用 cache： ```bash docker exec momo-pro-system python -c 'from services.cache_service import clear_all_cache; clear_all_cache(); print("cache_cleared")' ``` --- ## 14. 主機開機、關機、重啟 SOP 本節是每次 110 / 120 / 121 / 188 相關電源操作的標準程序。112 是 Kali，只保留 read-only evidence，不納入本輪恢復或例行重啟釋出。 ### 14.1 共同紅線 | 類型 | 禁止事項 | 正確處理 | |------|----------|----------| | 120 offline | 不可消音 `ColdStartHost120Unreachable`、`ColdStartRecoveryBlocked` 或 120 config backup alert | 保留紅燈，直到 console/VM recovery 後重跑完整 chain | | Filesystem | 不可對已掛載 root filesystem 做 online `fsck` | 只在 console/rescue/initramfs 狀態下離線修復 | | Backup | 不可用單項 backup 成功宣稱 aggregate backup green | 以 `backup-all`、offsite verifier、cold-start scorecard 三者共同判定 | | Credential | 不可把密碼、token、private key 寫進 repo、LOGBOOK 或聊天 | 只寫 non-secret evidence marker / vault reference | | Stateful data | 不可 truncate、DROP、整庫 restore 或整批刪 volume | 先保存證據，優先 `REINDEX TABLE CONCURRENTLY` / clean-clone / idempotent resync | | Automation | 不可在 P0/P1 未綠時釋出 runner/CD/AI full execution | observe-only，runner/CD 最後釋出 | ### 14.2 關機前 SOP 目標是保留證據、停止高負載來源、讓 stateful service 乾淨落地。 1. 宣告維護窗口，建立 `docs/LOGBOOK.md` 重啟紀錄草稿。 2. 跑 preflight snapshot： ```bash /home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1 /backup/scripts/backup-status.sh --no-notify /backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color ``` 3. 保存 host reboot evidence： ```bash for h in 110 120 121; do ssh wooo@192.168.0.$h 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20; systemctl --failed --no-pager' || true done ssh ollama@192.168.0.188 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20; systemctl --failed --no-pager' || true ``` 4. 暫停高負載與自動化釋出： | 順序 | 對象 | 操作原則 | |------|------|----------| | 1 | Gitea / actions runners | 停止新 job；不要在 build 中途硬關，先讓可完成 job 結束或人工取消 | | 2 | AI auto-remediation | 切 observe-only；禁止自動 restart stateful services | | 3 | momo crawler / scheduler / heavy batch | 暫停會啟動 Chrome、批次匯入或大量 DB 寫入的工作 | | 4 | Sentry/Snuba/ClickHouse heavy consumers | 確認沒有 restart storm；必要時 controlled stop | | 5 | K3s workload | 優先 drain / cordon 可達節點；不可在 120 已不可達時假裝 drain 完成 | 5. 全機關機順序： ```text 1. runner/CD and high-load batch 2. AI auto-remediation execution 3. AWOOOI workload layer 4. 121 K3s agent side 5. 120 K3s server side 6. 110 registry / observability, after evidence and backup status are captured 7. 188 data layer last 8. network / UPS / hypervisor last, if applicable ``` 188 必須最後關，因為 PostgreSQL / Redis / momo DB / K3s datastore 是其他層的共同依賴。 ### 14.3 開機 SOP 開機順序固定走 dependency chain，不追最吵的 alert。 ```text 1. Physical network: switch, NIC, ARP, SSH 2. 188 data layer: PostgreSQL, Redis, Docker, momo DB, SignOz dependencies 3. 110 registry / observability: Harbor, Gitea, Prometheus, Alertmanager, Sentry 4. 120 K3s server / VIP path 5. 121 K3s agent / failover path 6. AWOOOI API/Web workload 7. Public routes and Alertmanager E2E 8. Backups, cron, CronJobs, textfile exporters 9. momo scheduler / crawlers and high-load consumers 10. runners/CD 11. AI auto-remediation limited execution ``` 開機後每一層都要有 live evidence。最小驗收命令： ```bash for h in 110 120 121 188; do ping -c 2 -W 2 192.168.0.$h >/dev/null && echo "PING_OK 192.168.0.$h" || echo "PING_FAIL 192.168.0.$h" nc -G 3 -z 192.168.0.$h 22 && echo "SSH_OK 192.168.0.$h" || echo "SSH_FAIL 192.168.0.$h" done ssh ollama@192.168.0.188 'systemctl is-active docker postgresql@14-main redis-server nginx || true; pg_isready -h localhost -p 5432 || true; docker ps --format "{{.Names}}\t{{.Status}}" | head -80' ssh wooo@192.168.0.110 'systemctl is-active docker cron || true; curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true; curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true' ssh wooo@192.168.0.121 'sudo kubectl get nodes -o wide; sudo kubectl get pods -A | grep -v -E "Running|Completed" || true' /backup/scripts/backup-status.sh --no-notify /backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color /home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1 ``` ### 14.4 單主機重啟 SOP | Host | 重啟前條件 | 重啟後必查 | 完成條件 | |------|------------|------------|----------| | 110 | 不在 `backup-all` / rclone / verify window；runner job 已停止或人工取消；188 healthy | Docker, Harbor, Gitea, Prometheus, Alertmanager, Sentry, cron, textfile exporters, `/backup/scripts/backup-status.sh --no-notify` | 110 services green；backup status 沒有新增 stale / failed；runner/CD 最後釋出 | | 120 | 必須是 console-first 維護；若可達，先 cordon/drain；若不可達，不宣稱 drain 成功 | power/VM/NIC/boot/initramfs/fsck state, SSH, `kubectl get nodes`, `SchedulingDisabled` 清除狀態 | 120 ping/SSH OK；`mon Ready`；backup configs/all/offsite/verify/cold-start chain 重跑 | | 121 | 120 / 188 healthy；可達時先 cordon/drain | `k3s-agent` 或 live role、VIP 狀態、`kubectl get nodes`, pod placement | `mon1 Ready`；VIP / NodePort 路徑正常；workload 無新增 failed pods | | 188 | 110 backup status 已保存；停止或延後 momo heavy import；確認無 DB restore / migration | PostgreSQL, Redis, Docker, momo DB parity, SignOz/ClickHouse, cron, backup freshness | DB accepting；momo parity 綠；188 backup jobs fresh；高負載服務最後釋出 | ### 14.4.1 110 重啟後恢復指揮卡 110 是 registry / observability / backup center。重啟後先看 host 與核心端口，不要第一時間重啟 Docker daemon。 | 順序 | 檢查 | 成功基準 | 失敗處理 | |------|------|----------|----------| | 1 | `systemctl is-system-running` / failed units / Swap | `running`、failed `0` 或可解釋、Swap 未持續增加 | 先分辨 stale unit、現役 service、storage/network 問題 | | 2 | Docker daemon | `systemctl is-active docker=active` | 若 Docker `activating`，先看 journal；不要連續 restart/kill | | 3 | Harbor / registry | local `/v2/` 回 `200/401`，public registry 未登入 `401` | 只針對失效 upstream 做最小修復；避免 daemon restart | | 4 | Gitea / runners | Gitea 200/302；runner 最後釋出 | runner job 不可在 P0/P1 未綠時搶資源 | | 5 | Prometheus / Alertmanager | `/-/ready`、`/-/healthy` OK；required alerts visible | 若告警缺失，先修 rules/drift guard，再談自動化 | | 6 | Sentry / Langfuse / Stock / public tools | public 2xx/3xx；container 非 restart loop | 只修明確故障服務；不要 compose 全套重建 | | 7 | backup / offsite | `backup-status --no-notify`、offsite verifier | 120 不可達時 Configs 紅燈保留 | 110 post-reboot 最小命令： ```bash ssh wooo@192.168.0.110 ' date; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true free -h; swapon --show systemctl is-active docker cron || true curl -s -o /dev/null -w "harbor_v2=%{http_code}\n" --max-time 5 http://127.0.0.1:5000/v2/ || true curl -s -o /dev/null -w "gitea=%{http_code}\n" --max-time 5 http://127.0.0.1:3001/ || true curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true docker ps --format "{{.Names}}\t{{.Status}}" | head -120 ' ``` 2026-06-12 補充：`stockplatform-shared-ui-monitor.timer` 指向不存在的 legacy path 時，可停用 stale timer 解除 host failed unit；但正式 source-of-truth 必須後續清理，不能把 `reset-failed` 當修復。 ### 14.4.2 188 重啟後恢復指揮卡 188 是資料與 AI/Web 依賴主機。它恢復前，不釋出 K3s、AWOOOI API、momo heavy import 或 AI auto-remediation。 | 順序 | 檢查 | 成功基準 | |------|------|----------| | 1 | PostgreSQL | `pg_isready` accepting，無 checkpoint / WAL panic | | 2 | Redis | `PONG` | | 3 | Docker / containerd | active；momo-db / signoz / openclaw / litellm 非 restart loop | | 4 | momo DB parity | `daily_sales_snapshot` 與 `realtime_sales_monthly` 目前月份筆數與日期上下界一致 | | 4a | momo Google Drive token writeback | `/home/ollama/momo-pro/config/google_token.json` owner 對齊 Docker userns scheduler UID，mode 不寬於 `600`；不得讀取或輸出 token 內容 | | 4b | momo business data freshness | `daily_sales_snapshot` 最新日期落後 `0-2` 天可接受；落後 `3` 天以上為 `BLOCKED`，即使首頁 / health / DB parity 都正常也不可宣稱 full-stack green | | 5 | SignOz / monitoring bridge | HTTP 200；ClickHouse 不在修復風暴 | | 6 | momo scheduler | container healthy，recent activity pattern > 0；heavy import 等 DB green 後釋出 | | 7 | backup freshness | 188 backup textfile / 110 backup-from-188 freshness OK | 188 post-reboot 不可用「首頁 200」取代 DB parity，也不可用 DB parity 取代資料新鮮度。若出現 `posting list tuple ... cannot be split`，只走 `REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;`，不可 truncate 或整庫 restore。 2026-06-25 補充：若 `momo-scheduler` logs 出現 `Google Drive 認證失敗` / `could not locate runnable browser` / `Permission denied: 'config/google_token.json'`，先做 metadata-only 判讀，不得讀 token 內容。最新 10:35 readback 顯示 host path `/home/ollama/momo-pro/config/google_token.json` 與 container-side `config/google_token.json` 都是 missing，scheduler host UID 仍是 `100000`；因此不能沿用 2026-06-24「只改 owner/mode」的修復結論。解除 WARN 的最小安全流程是：取得 owner-provided non-secret evidence ref、確認維護窗口與 rollback owner、用不貼 token 的方式重新建立或恢復 token artifact、只檢查 `stat owner:group:mode` 與 scheduler auth readback、再跑 cold-start。未完成前，MOMO health 200 與 DB parity 不能取代 token/writeback evidence。 ### 14.4.3 120 恢復指揮卡 120 目前是 console-first blocker。它不可達時，遠端只能做證據收集，不能假裝修復。 | 狀態 | 判定 | 正確動作 | |------|------|----------| | ping / SSH / ARP 全失敗 | host / VM / network 層未知 | 到 hypervisor / console 確認 power、NIC、boot screen | | initramfs / fsck prompt | filesystem repair gate | 依 `120-fsck-maintenance-checklist.sh` 離線處理 | | SSH 恢復但 K3s NotReady | K3s / runtime 層 | 先看 `journalctl -u k3s`、containerd、188 PostgreSQL，再解除 cordon | | node Ready 但 SchedulingDisabled | 調度狀態未解除 | 確認健康後 `kubectl uncordon mon`，再看 workload | 120 恢復後不得只看 `kubectl get nodes`。必須強制補跑： ```bash /backup/scripts/backup-configs.sh /backup/scripts/backup-all.sh /backup/scripts/sync-offsite-backups.sh --mode sync /backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color /home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1 ``` ### 14.4.4 121 重啟後恢復指揮卡 121 是 K3s failover / secondary control-plane path。它重啟後的核心是「不要讓 mon1 Ready 掩蓋 mon 不可達」。 | 檢查 | 成功基準 | 注意 | |------|----------|------| | SSH / systemd | host ready、failed units 可解釋 | 121 green 不代表 120 green | | K3s role | `kubectl get nodes -o wide` 可讀 | 若只剩 `mon1 Ready`，仍是 degraded cluster | | VIP / NodePort | VIP / public routes 通 | 必須確認 route 走 `192.168.0.125:32334/32335` | | Cron / DR drill | cron present、DR drill 未被誤停 | schedule green 是 cold-start done criteria 的一部分 | 若 121 重啟後看到 `mon1 Ready` 但 `mon NotReady,SchedulingDisabled`，結論是「121 恢復，cluster 仍 degraded」，不可把 121 正常誤報成 K3s 全綠。 ### 14.5 每次重啟紀錄格式 每次開機、關機、重啟都要在 `docs/LOGBOOK.md` 追加紀錄，並把必要狀態同步到本 SOP 或 workplan。 ```text ## YYYY-MM-DD | Host reboot / shutdown / startup record Scope: - Hosts: - Operation: shutdown / startup / reboot / recovery - SOP version used: - Operator: - Maintenance window: Pre-check: - Cold-start scorecard: - Backup status: - Offsite verifier: - Public routes: - momo DB parity: - Alertmanager rules / E2E: - Credential escrow: Execution: - Start time: - End time: - Commands / console actions: - Services paused: - Services released: Result: - 110: - 120: - 121: - 188: - Cold-start scorecard after: - Backup status after: - Offsite verifier after: - DB parity after: - Alerts after: Difference versus previous reboot: - Faster: - Slower: - New blocker: - Repeated blocker: - False positive / detector tuning: - SOP change required: yes/no SOP update: - Previous version: - New version: - Change reason: - Files updated: ``` ### 14.6 SOP 版本比較與改版規則 每次重啟後必須比較上一次紀錄，不只寫「已恢復」。 | 比較項 | 判定方式 | |--------|----------| | Time to SSH | 從 power-on 到各 host SSH OK | | Time to K3s Ready | 從 120/121 boot 到 nodes Ready | | Time to public routes | 從 K3s Ready 到 public 2xx/3xx | | Time to backup green | 從 110 ready 到 backup status / offsite verifier green | | Persistent blockers | 連續兩次以上出現即列入 SOP hard gate | | False positives | 例如 momo scheduler detector WARN；要寫清楚直接證據與調整方向 | | Procedure drift | live cron、Ansible template、script path 與 SOP 不一致時，先修 canonical，再修 SOP | 改版規則： - 只更新 live baseline 或百分比：不升版，只更新日期與 evidence。 - 新增、刪除或改變操作順序：升 minor version，例如 `v1.4` -> `v1.5`。 - 牽涉破壞性操作、資料修復策略或人為批准邊界：升 major-ready review，先經人工批准。 ### 14.7 2026-06-06 重啟紀錄比較錨點 2026-06-06 沒有執行新重啟；本次是 live recovery check。它仍要作為下一次重啟比較基準： | 項目 | 2026-06-06 baseline | |------|----------------------| | Overall | `65% BLOCKED` | | Cold-start | `PASS=71 WARN=3 BLOCKED=3` | | Remaining hard blocker | 120 ping / SSH / K3s read-only check | | Backup aggregate | `failed=1`, Configs only, due 120 config capture | | Backup freshness | 110 and 188 fresh, no stale jobs | | Offsite | 13 repos latest-only green | | Escrow | 5 markers missing | | momo scheduler | direct healthy; 15:03 scorecard no longer emits scheduler WARN | ### 14.8 2026-06-12 重啟後比較錨點 2026-06-12 110 被非計畫重啟後，本 SOP v1.5 的新比較錨點如下： | 項目 | 2026-06-12 post-reboot baseline | |------|----------------------------------| | 110 host | `systemd running`，failed units `0`，Swap `0B/7.8GiB` | | 110 service recovery | Harbor / Gitea / Prometheus / Alertmanager / Sentry / Stock / public tools reachable | | Cold-start | `PASS=72 WARN=2 BLOCKED=3` | | Remaining hard blocker | 120 ping / SSH / K3s read-only check | | WARN | 120-driven backup aggregate/config component and 120 K3s schedule check | | Backup freshness | 110 `13/13 fresh failed=1`，188 `2/2 fresh failed=0`，stale none | | Offsite | 13 repos latest-only green，`REMOTE_LATEST_ONLY_OK=1`，`VERIFY_OK=1` | | Alerts | Prometheus and Alertmanager expose all five required backup/cold-start/escrow alerts | | momo scheduler | scorecard reads `SCHEDULER_RECENT_ACTIVITY 1070` after detector fix | | SOP change | v1.5 adds startup judgment layers, GO/NO-GO tree, host recovery cards, and timeline checks | ### 14.9 2026-06-13 CD 後恢復比較錨點 2026-06-13 不是主機重啟，而是用來驗證「120/121 workload balancing + CD known_hosts guardrail」是否能承受下一次正常部署的比較錨點。 | 項目 | 2026-06-14 03:10 baseline | |------|----------------------------| | Gitea / ArgoCD | Gitea main `8868c025`，deploy marker `7b034b58`，ArgoCD revision `8868c025`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `26b67d11f7b7de4f9c9d95c01bb1dacf4000e887` | | K3s placement | API/Web verified split across `mon` / `mon1` after the latest deploy marker；Worker single replica healthy | | Cold-start | `PASS=81 WARN=2 BLOCKED=0` | | Public routes | Scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS | | Backup | `backup-status`: 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5`，last aggregate `2026-06-14 02:40:22` | | Offsite | textfile `remote_verify_ok=1`、`full_verify_fresh=1`，13 repos each `snapshot_count=1` | | SSH trust | Global `known_hosts` retained 120 / 188 entries after CD; deploy-specific trust moved to `deploy_known_hosts` | | Remaining non-service debt | `km-vectorize-29689620` official Job failed with `BackoffLimitExceeded`; failed Pod/log was deleted before inspection; credential escrow missing count remains `5`; 110 has `fwupd` failed units | | SOP change | v1.10 changes the first-screen declaration from full green back to degraded, records official `km-vectorize` failure evidence, and verifies live `restartPolicy: Never` / `FallbackToLogsOnError` evidence retention for the next official run | ### 14.10 2026-06-14 110 failed-unit 清理比較錨點 2026-06-14 08:24 的變更不是主機重啟，而是把 110 非核心 `fwupd` failed-unit 噪音從 cold-start 判定中收斂。這個錨點的用途是避免未來把 firmware metadata refresh failure 誤判成 AWOOOI runtime 失敗，同時保留 rollback。 | 項目 | 2026-06-14 08:24 baseline | |------|----------------------------| | 110 failed units | `systemctl --failed` 回 `0 loaded units listed` | | fwupd policy | `fwupd-refresh.timer` 為 `disabled / inactive`，原因是非核心 firmware metadata refresh 失敗不應阻擋 AWOOOI service recovery | | Rollback | 若需要恢復 firmware metadata refresh timer，執行 `sudo systemctl enable --now fwupd-refresh.timer` 後重跑 cold-start | | Cold-start | `PASS=82 WARN=1 BLOCKED=0` | | Remaining WARN | 只剩 K8s failed Job `km-vectorize-29689620`；等待下一次官方 03:00 排程成功或保留 failed Pod/log | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | Credential escrow | 仍缺 5 個 non-secret evidence marker；不可用 placeholder 或 secret 清紅燈 | | SOP change | v1.11 把 110 failed-unit gate 從 `GREEN_WITH_FWUPD_WARNING` 改成 `GREEN_WITH_FWUPD_TIMER_DISABLED`，並把完成宣告上限固定為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.11 2026-06-14 post-CD recovery readback 2026-06-14 08:40 的變更不是主機重啟，而是確認 latest CD deploy marker 沒有讓重啟恢復狀態倒退。這個錨點用來比較「治理 / 前端 / API CD 後，cold-start SOP 是否仍成立」。 | 項目 | 2026-06-14 08:40 post-CD baseline | |------|------------------------------------| | Gitea / ArgoCD | Gitea main `18b867c3`，ArgoCD revision `18b867c3`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `e0a6d339669fc635357d36ea94215df25e652fa9` | | CronJob readback | `km-vectorize` has `KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z` | | K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` | | Cold-start | `PASS=82 WARN=1 BLOCKED=0` | | Public routes | Scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` | | Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` | | SOP change | v1.12 records the post-CD no-regression readback and keeps the declaration ceiling at `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.12 2026-06-14 P2-135 deploy 後 recovery readback 2026-06-14 09:27 的變更不是主機重啟，而是確認 P2-135 deploy 與正式驗證後，reboot recovery baseline 沒有倒退。這個錨點也記錄 stockplatform-v2 rollout warmup 期間短暫 `502` 的判定方式：直接重查 route / TLS，並重跑完整 cold-start；只有重跑仍失敗才升級成 persistent public route blocker。 | 項目 | 2026-06-14 09:27 post-P2-135 baseline | |------|----------------------------------------| | Gitea / ArgoCD | Gitea main `5bad267e`，ArgoCD revision `5bad267e`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `280e0fbef0d5dccb10f1efe2cc18cf423544254e` | | CronJob readback | `km-vectorize` has `KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z` | | K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon1` | | First cold-start | 09:26 first run saw `stock.wooo.work` `502` while stockplatform-v2 containers were less than one minute old; direct route and TLS recheck returned `200` | | Final cold-start | 09:27 rerun returned `PASS=82 WARN=1 BLOCKED=0` | | Public routes | Final scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` | | Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` | | SOP change | v1.13 records the P2-135 post-deploy no-regression readback and keeps the declaration ceiling at `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.13 2026-06-14 P2-136 / AI Agent 活動正式部署後 recovery readback 2026-06-14 09:56 的變更不是主機重啟，而是確認 P2-136 / AI Agent 活動正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點特別記錄 deploy marker、ArgoCD revision、live image 與 cold-start scorecard 必須一起看，避免只看 `gitea/main` 或 CD 成功就誤報 full-stack green。 | 項目 | 2026-06-14 09:56 post-P2-136 baseline | |------|----------------------------------------| | Gitea / ArgoCD | 本 recovery commit 前最新文件 head `a0fe7741`；runtime deploy marker `60a0415c chore(cd): deploy a3de0ff [skip ci]`，ArgoCD revision `60a0415c`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `a3de0ffb8275b6838604b6dff87cd978b8e91122` | | CronJob readback | `km-vectorize` has `KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` remains retained | | K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon1` | | Cold-start | 09:56 returned `PASS=82 WARN=1 BLOCKED=0` | | Public routes | Final scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` | | Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` | | SOP change | v1.14 records the P2-136 / AI Agent 活動正式部署後 no-regression readback and keeps the declaration ceiling at `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.14 2026-06-14 P2-137 / CI smoke timeout 修正後 recovery readback 2026-06-14 10:40 的變更不是主機重啟，而是確認 P2-137 正式部署與 BusyBox `timeout` smoke 修正後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P2-137 正式驗證內容。 | 項目 | 2026-06-14 10:40 post-P2-137 baseline | |------|----------------------------------------| | Gitea / ArgoCD | 本 recovery commit 前最新文件 head `50d4f2ba`；runtime deploy marker `d023f5d7 chore(cd): deploy f737f27 [skip ci]`，ArgoCD revision `50d4f2ba`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `f737f278dc14372ff1fb15b124b1370c20e1bb99` | | CronJob readback | `km-vectorize` has `KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` remains retained | | K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` | | Cold-start | 10:40 returned `PASS=82 WARN=1 BLOCKED=0` | | Public routes | Final scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` | | Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` | | SOP change | v1.15 記錄 P2-137 / CI smoke timeout 修正後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.15 2026-06-14 P2-143 owner response 預檢後 recovery readback 2026-06-14 15:00 的變更不是主機重啟，而是確認 P2-143 owner response 預檢與拒收邊界正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P2-142 / P2-143 正式驗證內容，也不把 owner response preflight 視為 runtime 授權。 | 項目 | 2026-06-14 15:00 post-P2-143 baseline | |------|----------------------------------------| | Gitea / ArgoCD | 最新文件基準 `b09eb1c6 docs(ai): 校準 P2-143 正式驗證紀錄`；runtime deploy marker `667d6329 chore(cd): deploy 755b0a8 [skip ci]`；ArgoCD revision `4abf0c0f750254d3c7137eae049abdfd99630f5f`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `755b0a8d3038df2c52dee280067863d92db1eda5` | | CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍保留，但目前沒有可讀的 failed Pod / log | | K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` | | Cold-start | 15:00 returned `PASS=82 WARN=1 BLOCKED=0` | | Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應 | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` | | P2-143 API boundary | Production endpoint 回 current `P2-143`、next `P2-144`、completion `100`，且 reviewer / Gateway queue、Telegram、Bot API、result capture、learning、PlayBook trust、production write、secret read、destructive operation 全部維持 `0 / false` | | Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` | | SOP change | v1.16 記錄 P2-143 owner response 預檢後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.16 2026-06-14 P2-144 owner response 回讀後 recovery readback 2026-06-14 15:58 的變更不是主機重啟，而是確認 P2-144 owner response 回讀狀態與後續 deploy marker `180a6543` 正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P2-144 正式驗證內容，也不把 owner response readback 視為 runtime 授權、正式收件或 owner acceptance。 | 項目 | 2026-06-14 15:58 post-P2-144 baseline | |------|----------------------------------------| | Gitea / ArgoCD | `gitea/main` 已前進至 `180a6543 chore(cd): deploy fef94df [skip ci]`；ArgoCD source revision `180a6543eaf26dd6b345d45114316926056a965a`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `fef94df877c5438f9f34ddbcace8ad8112a141ef` | | CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍保留，但目前沒有可讀的 failed Pod / log | | K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon1` | | Cold-start | 15:58 returned `PASS=82 WARN=1 BLOCKED=0` | | Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應 | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` | | P2-144 API boundary | Production endpoint 回 current `P2-144`、next `P2-145`、completion `100`，且 owner response received / accepted / rejected、reviewer / Gateway queue、Telegram、Bot API、result capture、learning、PlayBook trust、production write、secret read、destructive operation 全部維持 `0 / false` | | Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` | | SOP change | v1.17 記錄 P2-144 owner response 回讀後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.17 2026-06-14 P2-145 owner response 驗收門檻後 recovery readback 2026-06-14 16:29 的變更不是主機重啟，而是確認 P2-145 owner response 驗收門檻正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P2-145 正式驗證內容，也不把 acceptance gate 視為 owner response received / accepted、runtime 授權或正式寫入。 | 項目 | 2026-06-14 16:29 post-P2-145 baseline | |------|----------------------------------------| | Gitea / ArgoCD | 最新文件基準 `06fe0a8f docs(logbook): 記錄 P2-145 正式驗證 [skip ci]`；runtime deploy marker `36fbfc6b chore(cd): deploy 386dbd0 [skip ci]`；ArgoCD source revision `06fe0a8f14167824fea512f942d2569431bbcbc8`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `386dbd078ef63401d9736048463f4ef5326442d9` | | CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍為 `Failed 0/1` | | K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` | | Cold-start | 16:29 returned `PASS=82 WARN=1 BLOCKED=0` | | Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應 | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` | | P2-145 API boundary | Production endpoint 回 current `P2-145`、next `P2-146`、completion `100`，且 owner response received / accepted / rejected、reviewer / Gateway queue、Telegram、Bot API、result capture、learning、PlayBook trust、production write、secret read、destructive operation 全部維持 `0 / false` | | Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` | | SOP change | v1.18 記錄 P2-145 owner response 驗收門檻後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.18 2026-06-14 IwoooS P0 配置控管優先序後 recovery readback 2026-06-14 17:04 的變更不是主機重啟，而是確認 IwoooS P0 配置控管優先序正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P0 配置控管正式驗證內容，也不把前台看板可見視為 Nginx reload、DNS / TLS probe、certbot renew、workflow / secret 修改、public route change 或 runtime gate。 | 項目 | 2026-06-14 17:04 post-IwoooS-P0-config baseline | |------|--------------------------------------------------| | Gitea / ArgoCD | 最新文件基準 `af62ec1f docs(iwooos): 記錄 P0 配置控管正式驗證 [skip ci]`；runtime deploy marker `ed651a98 chore(cd): deploy e992af8 [skip ci]`；ArgoCD source revision `af62ec1fe72b3e84e179d80e788e5a5902bdaf27`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `e992af89955f8aae40a383b2f2e2f645445a690d` | | CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍為 `Failed 0/1` | | K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon1` | | Cold-start | 17:04 returned `PASS=82 WARN=1 BLOCKED=0` | | Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應；IwoooS route `/zh-TW/iwooos` 額外 readback 回 `200` | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` | | IwoooS boundary | P0 配置控管優先序已可見，但 live evidence received、runtime gate、Nginx live config、DNS / TLS probe、certbot renew、workflow / secret 修改、public route change、production write 仍不得從本 readback 推定為已授權 | | Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` | | SOP change | v1.19 記錄 IwoooS P0 配置控管優先序後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.20 2026-06-15 km-vectorize official success readback 2026-06-15 03:11 的變更不是主機重啟，而是確認 `km-vectorize` 官方 03:00 排程成功，並把 ArgoCD fully healthy gate 關閉。這個錨點只記錄 recovery / cold-start readback，不手動刪 Job、不手動建立 Job、不 `kubectl patch` live、不重啟服務，也不把任何 backup / restore / escrow owner acceptance ledger 視為 backup run、restore run、credential escrow marker write、host write 或 production write 授權。 | 項目 | 2026-06-15 03:11 km-vectorize official success baseline | |------|----------------------------------------------------------| | ArgoCD | `awoooi-prod` sync `Synced`，health `Healthy`，revision `d388e5b477333fd5e661527a729406a4e8215320` | | CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`lastScheduleTime=2026-06-14T19:00:00Z`、`lastSuccessfulTime=2026-06-14T19:00:55Z` | | Job / Pod / log | Job `km-vectorize-29691060` `Complete`，Pod `km-vectorize-29691060-78xpz` `Completed` restart `0`，log `embed-all: 200 {"total":31,"success":31,"failed":0}` | | Cold-start | 03:11 returned `PASS=81 WARN=2 BLOCKED=0`，result `DEGRADED` | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，last aggregate `2026-06-15 02:40:13` | | Escrow | `ESCROW_MISSING_COUNT=5`，缺 `restic_repository_password`、`offsite_provider_credentials`、`break_glass_admin_credentials`、`dns_registrar_recovery`、`oauth_ai_provider_recovery` | | Remaining warnings | 188 momo scheduler registration/activity 未確認；K8s 仍保留舊 failed Job evidence | | SOP change | v1.21 關閉 `km-vectorize` official success gate，但宣告上限仍是 `SERVICE_AVAILABLE_ARGOCD_HEALTHY_DR_ESCROW_BLOCKED`；不可宣稱 `full-stack green` 或 `DR complete` | ### 14.19 2026-06-14 高價值配置 Owner Packet 前台同步後 recovery readback 2026-06-14 18:15 的變更不是主機重啟，而是確認高價值配置 Owner Packet 前台同步正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 Owner Packet 前台正式驗證、posture projection 或 intake preflight 內容，也不把前台草案可見視為 request sent、owner response received / accepted、runtime gate、Nginx reload、DNS / TLS probe、certbot renew、workflow / secret 修改、host write、active scan 或 production write。 | 項目 | 2026-06-14 18:15 post-owner-packet-frontend baseline | |------|-------------------------------------------------------| | Gitea / ArgoCD | 最新 repo 文件基準 `0a4766dd docs(security): 新增高價值配置 owner request 草稿包 [skip ci]`；runtime deploy marker `16c6b983 chore(cd): deploy e999c16 [skip ci]`；feature commit `e999c16b fix(iwooos): 同步高價值配置 owner packet 前台`；ArgoCD source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94`，sync `Synced`，health `Degraded` | | K3s image readback | API/Web/Worker/CronJob image tag `e999c16b3435f197b78fe2adfeec1c4faa6c4675` | | CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍為 `Failed 0/1` | | K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` | | Cold-start | 18:15 returned `PASS=82 WARN=1 BLOCKED=0` | | Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應；IwoooS route `/zh-TW/iwooos` 與 AwoooP route `/zh-TW/awooop` 額外 readback 皆回 `200` | | Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` | | 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` | | Owner Packet boundary | Owner Packet 前台數字已可見，但 request sent、owner response received / accepted / rejected、reviewer queue write、live evidence、runtime gate、Nginx live config、DNS / TLS probe、certbot renew、workflow / secret 修改、host write、active scan、production write 仍不得從本 readback 推定為已授權 | | Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` | | SOP change | v1.20 記錄高價值配置 Owner Packet 前台同步後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` | ### 14.21 2026-06-18 Plan B 降級運轉路徑 2026-06-18 的變更不是主機重啟，也不是新的 live recovery readback，而是把統帥要求的 Plan B 明確寫成可執行 SOP。這個錨點用來比較下一次重啟時是否有照 §1.4 先判斷 Plan A / Plan B、降級等級、停止線與回到 Plan A 的條件。 | 項目 | 2026-06-18 Plan B baseline | |------|----------------------------| | SOP version | `v1.22` | | Plan B trigger | backup/offsite/verifier running、P0 host 15 分鐘不可達、188 data unhealthy、110 registry / observability unhealthy、單台 K3s degraded、route-only green、cold-start WARN、credential escrow missing | | Service levels | `B0_ABORTED_BEFORE_REBOOT`、`B1_HOST_RECOVERY_ONLY`、`B2_CORE_SERVICE_READY`、`B3_SERVICE_AVAILABLE_DEGRADED`、`B4_FULL_STACK_GREEN`、`B5_DR_COMPLETE` | | Host fallback paths | 110 / 120 / 121 / 188 / K3s / Public gateway 各自有降級路徑與回到 Plan A 的條件 | | Timeline | `T+0` freeze、`T+5` host boot、`T+15` data / registry stop-line、`T+30` route-only guard、`T+60` cold-start scorecard、`T+120` incident / follow-up | | Closeout states | `RETURNED_TO_PLAN_A`、`SERVICE_AVAILABLE_DEGRADED`、`OPEN_INCIDENT_REQUIRED` | | SOP change | v1.22 新增 Plan B；不可把 Plan B 視為 runtime write 授權，也不可因文件化 Plan B 宣稱新的 service green、full-stack green 或 DR complete | ### 14.23 2026-06-18 repo-side readiness audit blocker closure 2026-06-18 的第二段變更不是 live recovery，也不是主機重啟，而是把前一輪 readiness audit 的 repo-side hard blockers 收斂成可驗證契約。這個錨點代表「重啟 SOP / baseline / scripts / Ansible source-of-truth / Gitea workflow contract 在 repo 內已可通過 readiness audit」，不代表當日 live hosts 已重新驗證。 | 項目 | 2026-06-18 repo-side readiness baseline | |------|-----------------------------------------| | SOP version | `v1.23` | | Cold-start gate | `full-stack-cold-start-check.sh` 新增 `NODE_FS_ERROR_EVENTS`，120 / K3s node event 出現 filesystem / fsck / read-only / I/O 類證據時，不能宣稱 K3s safe | | Backup contract | `backup-awoooi.sh` 移除 service-level 直接 offsite sync；offsite 發布只走集中 `sync-offsite-backups.sh` / verifier gate | | Ansible 110 source-of-truth | `110-devops.yml` 納入 cold-start monitor、runner guardrails、host textfile exporters、backup scripts、daily backup heartbeat、offsite evidence report、offsite full-sync verifier | | Ansible 188 source-of-truth | `188-ai-web.yml` 納入 textfile exporters，並把 momo PostgreSQL backup entrypoint 固定到 host-owned `/home/ollama/bin/momo-pg-backup.sh` | | Nginx source-of-truth | `nginx-sync.yml` 納入 `188-internal-tools-https.conf.j2` route sync | | CI / workflow contract | `.gitea/workflows/ansible-lint.yml` 改為 self-hosted validation，觸發範圍包含 Ansible、ops baseline、monitoring rules、backup scripts、reboot scripts、docs 與 workflow 自身 | | Validation toolchain | `bootstrap-ansible-validation-env.sh` 會優先使用 Python 3.11 / 3.10 建立 pinned validation venv；`ansible-validate.sh` 固定 repo roles path，並以 minimum lint profile 守住 syntax / loader readiness | | Repo-side readiness audit | `PASS=185 WARN=1 BLOCKED=0`，結果 `READY WITH WARNINGS`；唯一 warning 是未跑 `--live` | | Declaration limit | 可宣稱 `REPO_SIDE_REBOOT_READINESS_READY_WITH_LIVE_CHECK_REQUIRED`；不可宣稱 `FULL_STACK_GREEN`、`DR_COMPLETE` 或 live service recovery complete | ### 14.24 2026-06-18 live cold-start readback after repo-side closure 2026-06-18 12:13-12:17 的 readback 是 repo-side readiness closure 後的同日 live 驗證。這不是主機重啟，也不是 runtime 修復；它的用途是把「機制已完成」和「當下 live 狀態」分開，避免 false-green。 | 項目 | 2026-06-18 12:17 live baseline | |------|--------------------------------| | SOP version | `v1.24` | | Cold-start read-only result | `PASS=83 WARN=1 BLOCKED=0`，result `DEGRADED` | | Host reachability | 110 / 120 / 121 / 188 ping OK and SSH port OK | | K3s | `mon` / `mon1` Ready control-plane；VIP `192.168.0.125` present on 120；`NODE_FS_ERROR_EVENTS 0` | | 110 / 188 service checks | 110 Harbor / Gitea / Prometheus / Alertmanager / Sentry reachable；188 PostgreSQL / Redis / momo / SigNoz reachable | | Backup health | 110 backup health `total=13 stale=0 missing_cron=0 missing_script=0 failed_count=0 config_failed=0 integrity_total=2 integrity_stale=0`；188 backup health `total=2 stale=0` | | Public route / TLS | awoooi API/Web、mo、momo health、Gitea、Harbor、registry、Sentry、SigNoz、stock、Langfuse、Bitan、aiops all 2xx/3xx with TLS verified | | AWOOOI rollout convergence | After transient 12:14 startup window, final readback shows API `2/2`, Web `2/2`, Worker `1/1`, Canary `1/1`, API health `200 healthy` | | Remaining warning | retained stale Job `km-vectorize-29689620` from 2026-06-14 03:00; later official Jobs `km-vectorize-29692500`, `29693940`, `29695380` are `Complete` | | Declaration limit | 可宣稱 `SERVICE_AVAILABLE_DEGRADED`；不可宣稱 `FULL_STACK_GREEN`，因為 `WARN=1`；不可宣稱 `DR_COMPLETE`，credential escrow evidence still requires real non-secret owner evidence | ### 14.25 2026-06-18 stale failed Job classification and service-green readback 2026-06-18 13:43 的變更不是刪除 K8s Job，也不是手動建立 Job，而是修正 cold-start 判定邏輯：保留的歷史 failed Job 是 evidence；只有沒有後續官方成功 Job 的 failed Job 才是 active blocker。這讓 evidence retention 和 service readiness 不再互相衝突。 | 項目 | 2026-06-18 13:43 stale Job classification baseline | |------|----------------------------------------------------| | SOP version | `v1.25` | | Script change | `full-stack-cold-start-check.sh` emits `FAILED_JOBS`, `STALE_FAILED_JOBS`, and `ACTIVE_FAILED_JOBS` | | Active blocker rule | `ACTIVE_FAILED_JOBS > 0` causes warning; `STALE_FAILED_JOBS > 0` is retained evidence and does not warn by itself | | Readiness audit contract | `reboot-recovery-readiness-audit.sh` requires both stale and active failed Job counters | | Repo-side validation | `bash -n` passed; readiness audit returned `PASS=187 WARN=1 BLOCKED=0` with only the expected non-live warning | | 110 live script sync | `/home/wooo/scripts/full-stack-cold-start-check.sh` hash `b48af9c603aa5a1a4f9434d6cc510398bbecc2e46400a21410e735d5f7d177c4`; previous version backed up to `/home/wooo/scripts/full-stack-cold-start-check.sh.before-stale-active-job-classification.20260618-135516` | | Live cold-start readback | `PASS=84 WARN=0 BLOCKED=0`, result `GREEN` | | K8s Job evidence | `FAILED_JOBS=1`, `STALE_FAILED_JOBS=1`, `ACTIVE_FAILED_JOBS=0`, `BAD_PODS=0` | | Backup / DR evidence | 110 backup health `13/13 fresh failed=0`; 188 backup health `2/2 fresh failed=0`; escrow readback still `ESCROW_MISSING_COUNT=5` | | Declaration limit | 可宣稱 `FULL_STACK_GREEN_FOR_SERVICE`；不可宣稱 `DR_COMPLETE`、`credential escrow complete` 或任何 runtime/security acceptance | | SOP change | v1.25 defines retained failed Job evidence vs active failed Job blocker; future reboot comparison must record all three counters | ### 14.26 2026-06-24 heartbeat noise / MOMO detector / rollout false-negative closure 2026-06-24 的變更不是主機重啟，而是把重啟 SOP 的兩種 false signal 收斂：Telegram 正常心跳不再每 30 分鐘洗版；MOMO scheduler / current-month parity detector 不再因舊 log pattern 或錯誤 DB exec 使用者誤報 WARN。這個錨點也記錄 CD rollout false-negative：worker startup probe 第一次超時重啟一次，K8s 最終 ready，但 Gitea CD `#3289` 因 rollout status timeout 標 Failure。 | 項目 | 2026-06-24 live baseline | |------|--------------------------| | SOP version | `v1.27` | | Heartbeat code | `a84a5a0b fix(api): suppress healthy Telegram heartbeat noise` | | Deploy marker | `4a7b5329 chore(cd): deploy a84a5a0 [skip ci]` | | Production image readback | API/Web/Worker image tag `a84a5a0bc4a672ac6feb95a85ac590aa2dd4bb71` | | Production rollout | API `2/2`、Web `2/2`、Worker `1/1` Ready | | CD result caveat | Gitea CD `#3289` shows Failure because worker rollout status timed out before old replica convergence; K8s deploy marker and production readiness are green | | Healthy heartbeat rule | `status=healthy` 且無 warnings 時只更新 suppression marker / log，不送 Telegram；warnings 與 recovery 仍可送 | | Live temporary suppression | Redis keys `heartbeat:silent_last_sent` and `heartbeat:healthy_suppressed_last_seen` set with 24h TTL during deployment; no token or secret printed | | 110 live script sync | `/home/wooo/scripts/full-stack-cold-start-check.sh` hash `47e67d0c018f741acfba17a93cb1d668779bd08745902099a10ee61e73ea55b6`; previous version backed up to `/home/wooo/scripts/full-stack-cold-start-check.sh.before-momo-detector-20260624-020759` | | MOMO scheduler evidence | `SCHEDULER_CONTAINER_RUNNING true`、`SCHEDULER_CONTAINER_HEALTH healthy`、`SCHEDULER_RECENT_ACTIVITY 1303` | | MOMO DB parity evidence | `MOMO_MONTHLY_SYNC 10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17` | | K3s node evidence | `NODE_FS_ERROR_EVENTS 0`、`NODE_READONLY_FILESYSTEM_TRUE 0`、`NODE_DISK_PRESSURE_TRUE 0`、VIP `192.168.0.125` present | | Live cold-start readback | `PASS=85 WARN=0 BLOCKED=0`, result `GREEN` | | Declaration limit | 可宣稱 current service recovery scorecard green；不可宣稱 `DR_COMPLETE`，credential escrow evidence missing remains `5` | | SOP change | v1.27 requires heartbeat success-message suppression, MOMO detector parity using app-provided DB env, and rollout false-negative classification before retrying CD | Worker / CronJob / queue 類服務若啟動時間可能超過 startup probe，不能只看第一次 `rollout status --timeout=60s` 失敗就判定 production down。必須同時看 deploy marker、image tag、pod readiness、container restart count、service health、public route / API health。若 pod 最終 ready 但 CD 紅燈，這是 CI timeout / probe tuning 工作，不是服務重啟事故；後續應調整 startup probe 或 post-deploy timeout。 2026-06-24 02:44 補充：本節的 02:08 `PASS=85 WARN=0 BLOCKED=0` 已被 §14.28 的 MOMO data freshness gate 取代；不可再引用該結果宣稱 full-stack green。 ### 14.27 2026-06-24 188 node-exporter / backup health alert closure 2026-06-24 的第二段變更是恢復 188 node-exporter textfile scrape。`backup-status` 與 cold-start 都能透過 SSH 讀到 188 `backup_health.prom` fresh，但 Prometheus `node-exporter-188` scrape down 會讓 `BackupHealthMonitorMissing188` 正確告警。這種情況不能消音告警，必須恢復 exporter。 | 項目 | 2026-06-24 188 exporter baseline | |------|----------------------------------| | SOP version | `v1.28` | | Root cause | 188 `9100` connection refused；`node_exporter` / `prometheus-node-exporter` unit absent/inactive；Prometheus could not scrape `backup_health.prom` | | False start | Mounting `/home/ollama/node_exporter_textfiles` via `/host/home/ollama/...` failed because `/home/ollama` is `750` and textfile collector saw `permission denied` | | Live restore | Docker container `node-exporter` uses `quay.io/prometheus/node-exporter:v1.8.2`, `restart=unless-stopped`, `-p 9100:9100`, rootfs mount `/host`, direct textfile bind `/home/ollama/node_exporter_textfiles:/textfile:ro` | | Repo helper | `scripts/ops/188-node-exporter-restore.sh` | | Local metrics | `awoooi_backup_health_monitor_up{host="188"} 1`; `node_textfile_scrape_error 0` | | Prometheus readback | `up{job="node-exporter-188"} 1`; `awoooi_backup_health_monitor_up{host="188"} 1`; `absent(awoooi_backup_health_monitor_up{host="188"})` empty | | Alert readback | `ALERTS{alertname="BackupHealthMonitorMissing188"}` empty | | Declaration limit | 可宣稱 188 backup health scrape restored；不可把這當作 credential escrow complete 或 backup retention policy complete | 若未來重啟後 `BackupHealthMonitorMissing188` active，但 SSH/backup-status 顯示 `backup_health.prom` fresh，優先查： ```bash curl -fsS http://192.168.0.188:9100/metrics | grep -E 'awoooi_backup_health_monitor_up|node_textfile_scrape_error' ``` 若 `9100` connection refused 或 textfile collector error，先用 repo helper 恢復 exporter： ```bash ssh ollama@192.168.0.188 'bash -s' < scripts/ops/188-node-exporter-restore.sh ``` 恢復後再查 Prometheus / Alertmanager，不要直接 silence。 ### 14.28 2026-06-25 MOMO Google Drive token 與資料新鮮度 blocker 2026-06-24 的第三段變更是把「MOMO 服務活著但資料不新」納入 cold-start hard gate。2026-06-25 11:44 曾證明 MOMO 服務、public route、DB parity、scheduler activity、backup/offsite 都可用，但 Google Drive token artifact metadata missing 且資料停在 `2026-06-17`，所以 cold-start 正確 BLOCKED。2026-06-25 14:16 的最新狀態已由合法匯入 job `57` 解除該資料新鮮度 blocker：MOMO service health 是 `V10.674`，`daily_sales_snapshot` 與 `realtime_sales_monthly` 皆到 `2026-06-24`，`MOMO_DAILY_FRESHNESS 1|2026-06-24`，dedicated preflight `PASS=18 WARN=3 BLOCKED=0`。這仍不代表 DR complete，也不代表可以讀取或保存 Google Drive token 內容。 | 項目 | 2026-06-25 MOMO freshness / token baseline | |------|------------------------------------| | SOP version | `v1.51` | | Token current state | `MOMO_GDRIVE_TOKEN_STAT 100000:100000:600 scheduler_uid=100000`; dedicated preflight also saw host token metadata aligned to scheduler UID and container-side token artifact mode `600`; token content was not read | | Token recovery boundary | Owner-gated maintenance only；不得讀取、貼上、保存 token value / hash / partial；不得把聊天密碼或 workaround 寫進 repo | | Drive auth behavior | 2026-06-25 10:04 fail-closed evidence remains historical proof that auth failure does not become a fake success. 14:16 readback shows the later legitimate import succeeded and the blocker is cleared. | | Drive pending folder | `當日業績匯入`，pattern `即時業績_當日`; latest successful source recorded by job `57` | | Latest valid import | Job `57 completed`，`即時業績_當日.xlsx`，`2026-06-25T13:16:47.359958..2026-06-25T13:18:02.964985`，`15383/15383/0` | | DB parity | `daily_sales_snapshot=109061|2025-07-01|2026-06-24`; cold-start `MOMO_MONTHLY_SYNC 15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24` | | Data freshness | `MOMO_DAILY_FRESHNESS 1|2026-06-24` as of 2026-06-25 14:16 | | Live cold-start readback | `PASS=89 WARN=0 BLOCKED=0`, result `GREEN`; dedicated MOMO preflight `PASS=18 WARN=3 BLOCKED=0` | | 110 live script sync | `/home/wooo/scripts/full-stack-cold-start-check.sh` hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8` | | Alert behavior | Drive auth failure must send failure notification; heartbeat success remains suppressed; stale data alert should clear only with fresh DB evidence like job `57` / freshness `1` | | Declaration limit | 可宣稱 hosts/routes/K3s/backups/MOMO service/MOMO data freshness recovered for this evidence set；不可宣稱 DR complete、credential escrow complete、Wazuh host registry accepted 或 runtime/security acceptance | MOMO post-reboot 最小 readback： ```bash scripts/reboot-recovery/momo-drive-token-source-recovery-preflight.sh ssh ollama@192.168.0.188 ' stat -c "%u:%g:%a %n" /home/ollama/momo-pro/config/google_token.json 2>/dev/null || echo "google_token.json missing" docker top momo-scheduler -eo pid,user,uid,gid,args | head -n 3 docker logs --since 2h momo-scheduler 2>&1 | grep -E "AutoImport|Google Drive|Permission denied|could not locate runnable browser|沒有找到|發現檔案|匯入失敗通知" | tail -120 ' ssh ollama@192.168.0.188 'db_user=$(docker exec momo-pro-system printenv POSTGRES_USER); db_name=$(docker exec momo-pro-system printenv POSTGRES_DB); db_pass=$(docker exec momo-pro-system printenv POSTGRES_PASSWORD); docker exec -i -e PGPASSWORD="$db_pass" momo-db psql -h 127.0.0.1 -U "$db_user" -d "$db_name" -At' <<'SQL' SELECT 'daily_sales_snapshot|' || count(*) || '|' || min(snapshot_date)::date || '|' || max(snapshot_date)::date FROM daily_sales_snapshot; SELECT 'realtime_sales_monthly|' || count(*) || '|' || min("日期")::date || '|' || max("日期")::date FROM realtime_sales_monthly; SELECT 'daily_freshness|' || (CURRENT_DATE - max(snapshot_date)::date) || '|' || max(snapshot_date)::date FROM daily_sales_snapshot; SQL ``` Preferred path is the scripted preflight. It is read-only and returns `0` for clean, `1` for WARN-only, and `2` for BLOCKED. 2026-06-25 14:16 live run returned `PASS=18 WARN=3 BLOCKED=0`: `https://mo.wooo.work/health` and local health both returned `200`, health version was `V10.674`, app / scheduler / Telegram bot were healthy, scheduler restart count was `0`, token metadata aligned to scheduler UID without reading token content, current-month DB parity matched, latest daily import job `57` was clean, and `DB_DAILY_FRESHNESS 1|2026-06-24` cleared the MOMO hard blocker. The remaining WARNs are stability / future-evidence notes, not blockers. 若 Drive token artifact missing 或 Drive pending folder 無新來源檔，不可手動 truncate、不可以舊 archive 檔重複匯入來製造「最新」，也不可把 DB parity 當 data freshness。下一個解除 blocker 的證據必須是： 1. Owner 提供非 secret evidence ref，確認可以恢復 Google Drive token artifact 或合法來源檔。 2. 維護窗口、rollback owner、post-check owner 明確記錄。 3. token artifact 只用 metadata 驗證：owner 對齊 scheduler UID、mode 不寬於 `600`、不輸出 token 內容。 4. 新的 `即時業績_當日` source file 可見，或 scheduler 能成功列出待匯入來源。 5. import job 成功，`sync_success=true`，且 Drive 檔案只在成功後移動。 6. `daily_sales_snapshot` 與 `realtime_sales_monthly` 日期上下界一致，且 `MOMO_DAILY_FRESHNESS <= 2`。 ### 14.29 2026-06-24 188 MinIO / Velero、DB exporter 與 110 disk pressure recovery 2026-06-24 的第四段變更是恢復真正的備份與監控鏈路，而不是消音告警。`VeleroBackupNotRun`、`PostgreSQLDown`、`RedisDown`、110 disk pressure 都是有效紅燈；修復順序必須是 source-of-truth / service / exporter / Prometheus / Alertmanager / cold-start scorecard。 | 項目 | 2026-06-24 06:35 recovery baseline | |------|-------------------------------------| | SOP version | `v1.30` | | 188 DB exporter root cause | Docker user namespace 下 exporter compose 不能使用 `network_mode: host`；Redis live port 是 `6380` | | 188 DB exporter source-of-truth | `ops/monitoring/docker-compose.exporters.yaml` 改為 bridge port mapping；PostgreSQL DSN 只從 host `.env.exporters` 注入，repo 不放密碼 | | 188 DB exporter helper | `scripts/ops/188-db-exporters-restore.sh`；live path `/home/ollama/bin/188-db-exporters-restore.sh` | | 188 DB exporter readback | local metrics `pg_up=1`、`redis_up=1`；Prometheus `up{job="postgres-exporter"}=1`、`pg_up=1`、`up{job="redis-exporter"}=1`、`redis_up=1` | | 110 disk pressure | `/` from `92%` used to `73%` used after Docker image / build cache cleanup only; no Docker volume prune | | MinIO / Velero root cause | 188 MinIO endpoint `192.168.0.188:9000` was down; Velero BSL S3 list failed; MinIO data path had userns write denial | | MinIO restore | live `/home/ollama/minio/docker-compose.override.yml` adds `userns_mode: host` for the `minio` service; MinIO health endpoint is OK | | Velero restore | 120 `BackupStorageLocation/default` phase is `Available`; one-off backup `reboot-recovery-202606240456` is `Completed` | | Backup-health textfile | 110 exporter refresh reports `awoooi_velero_monitor_up=1`, `awoooi_velero_latest_completed_backup_fresh=1`, restore-test cron present, failed jobs `0` | | Alert readback | `VeleroBackupNotRun`、`PostgreSQLDown`、`RedisDown`、110 disk-pressure alerts resolved | | Live cold-start readback | `PASS=86 WARN=0 BLOCKED=1`, result `BLOCKED`; only blocker remains MOMO data freshness | | Declaration limit | 可宣稱 backup / exporter / MinIO / Velero chain recovered；不可宣稱 full-stack green、MOMO data current、DR complete 或 runtime/security acceptance | 188 PostgreSQL / Redis exporter post-reboot recovery: ```bash ssh ollama@192.168.0.188 'bash /home/ollama/bin/188-db-exporters-restore.sh' curl -fsS http://192.168.0.188:9187/metrics | grep '^pg_up ' curl -fsS http://192.168.0.188:9121/metrics | grep '^redis_up ' ``` 188 MinIO / 120 Velero recovery from 110: ```bash ssh wooo@192.168.0.110 '/home/wooo/scripts/188-minio-velero-restore.sh' ``` 如果需要在維護窗口中建立一次性 reboot-recovery 備份並刷新 110 backup-health textfile，必須明確設定： ```bash ssh wooo@192.168.0.110 'CREATE_VELERO_BACKUP=true REFRESH_BACKUP_HEALTH=true /home/wooo/scripts/188-minio-velero-restore.sh' ``` 本地 repo helper 可同步 live script： ```bash scp -q scripts/ops/188-db-exporters-restore.sh ollama@192.168.0.188:/home/ollama/bin/188-db-exporters-restore.sh scp -q scripts/ops/188-minio-velero-restore.sh wooo@192.168.0.110:/home/wooo/scripts/188-minio-velero-restore.sh ``` 110 disk pressure cleanup rule: ```text Allowed in incident recovery: Docker image / build cache cleanup after checking `docker system df`. Forbidden without explicit owner approval: `docker volume prune`, deleting database / registry / MinIO / ClickHouse / Sentry / PostgreSQL volumes, or removing unknown bind-mounted state. Done gate: filesystem use below 85%, no active disk-pressure alerts, and no service regression in cold-start scorecard. ``` ### 14.30 2026-06-24 notification noise closure after reboot recovery 2026-06-24 的第五段變更是把「服務已恢復，但舊監控路徑或成功心跳繼續洗 Telegram」納入重啟 SOP。這不是消音；失敗、warning、資料新鮮度、backup / exporter / escrow 紅燈仍要告警。修正目標是避免同一個已知失敗每 5 或 30 分鐘重複推送，並避免正常成功心跳佔滿戰情室。 | 項目 | 2026-06-24 notification baseline | |------|-----------------------------------| | SOP version | `v1.31` | | AWOOOI healthy heartbeat | Production `a84a5a0b`：healthy 且無 warnings 時只更新 Redis/log，不送 Telegram；warning 變化會送，warning 恢復 healthy 只送一次 recovery | | MOMO false-noise root cause | 110 舊 `/home/wooo/scripts/docker_health_monitor.sh` 打 `http://192.168.0.188/health`，重啟期間連續得到 `HTTP 502`，產生每 5 分鐘 MOMO Pro 告警 | | MOMO monitor source-of-truth | 新增 `scripts/ops/momo-pro-health-monitor.sh`；primary truth 是 `https://mo.wooo.work/health`，188 local `127.0.0.1:5003/health` 與 container state 只作 secondary evidence | | MOMO live readback | `/home/wooo/scripts/docker_health_monitor.sh` hash `d7a6bc75549efa10176c42e6f9082c90b9856dbcbb335aba4a4fa4abb754eaef`; manual run returned `OK: public health 200; no alert` | | AWOOI ops notify wrapper | `/home/wooo/awoooi-ops/notify-awoooi-ops.sh` hash `12bf9ae124c56bb7f31be15ebeb501671b0686d695492bc3fa1d9abb5b683b67`; repo MOMO monitor uses this wrapper instead of adding a new Telegram Bot API direct send | | Docker monitor fallback | `scripts/ops/docker-health-monitor.sh` keeps `ACTION_COOLDOWN_SECONDS=300` for repair cadence but adds `NOTIFY_COOLDOWN_SECONDS=1800` for direct Telegram fallback when AWOOOI API cannot receive the event | | Docker monitor live readback | `/home/wooo/awoooi-ops/docker-health-monitor.sh` hash `41d64f29048868c8e4c089132d299c8ee0e2b50ab2c513158d6d45cf92ea38e3` and exposes `TELEGRAM_COOLDOWN` lines for repeated fallback suppression | | Bitan public-content check | Live `/home/wooo/apps/bitan-pharmacy-release/scripts/run-public-content-cleanliness-check.sh` now writes `public-content-cleanliness.notify.state`, suppresses same failure fingerprint for `21600s`, and sends one recovery notice after a failed state becomes pass | | Bitan live readback | Script hash `294ec7f75448c86688b8afc408c785efe4cf173d468ad0d82228ba638d3de2dc`; manual no-notify run returned PASS for DB, public APIs, products/news pages, and content contract | | Declaration limit | 可宣稱 repeated healthy / same-failure notification noise is controlled for these paths；不可宣稱 all product alerts migrated to the unified notification gateway or any real failure alert disabled | Post-reboot notification gate: ```bash ssh wooo@192.168.0.110 '/home/wooo/scripts/docker_health_monitor.sh' ssh wooo@192.168.0.110 'tail -n 120 /home/wooo/logs/docker_health.log' ssh wooo@192.168.0.110 'tail -n 120 /home/wooo/awoooi-ops/monitor.log' ssh wooo@192.168.0.110 'tail -n 120 /home/wooo/apps/bitan-pharmacy-release/logs/public-content-cleanliness-check.cron.log' ``` Done gate: ```text MOMO monitor: public health 200 -> no Telegram. AWOOOI heartbeat: healthy + no warnings -> suppressed; warning/recovery still send. Generic docker-health monitor: API 200/202 path is primary; direct Telegram fallback is fingerprint-cooled. Bitan public content: pass -> no failure Telegram; repeated same failure -> cooled; recovery after prior failure -> one notice. ``` ### 14.31 2026-06-24 MOMO source-file absence decision gate 2026-06-24 11:35 的恢復判定把 MOMO 分成兩件事：服務可用與資料新鮮。服務可用已恢復，資料新鮮仍 blocked。這個 gate 的目的，是防止 operator 在外部網站 200、container healthy、DB parity 正常時，誤把「沒有新來源檔」當成「恢復完成」。 | 項目 | 11:35 source-file absence baseline | |------|------------------------------------| | SOP version | `v1.32` | | MOMO public health | `https://mo.wooo.work/health` returns healthy; version `V10.639` | | DB rows | `daily_sales_snapshot=104614`，`realtime_sales_monthly=786621` | | DB bounds | daily `2025-07-01..2026-06-17`；monthly `2024-01-01..2026-06-17` | | Current-month parity | `10936|10936|2026-06-01|2026-06-17|2026-06-01|2026-06-17` | | Latest successful import | `daily_sales` job `56`，created `2026-06-18 11:41`，source `即時業績_當日.xlsx`，`sync_success=true` | | Pending source folder | `當日業績匯入` count `0` for pattern `即時業績_當日` | | Archive latest | `2026-06-18T01:30:39Z`，already imported by job `56` | | Scheduler Drive readback | container-side Drive listing works and currently returns count `0`; no current `Permission denied` evidence in latest readback | | Stale alert posture | `data_stale_alert` has 24h dedupe; this is a true warning, not heartbeat spam | | Blocking metric | `MOMO_DAILY_FRESHNESS 7|2026-06-17` | | Repo-side v1.42 scorecard evidence | `MOMO_SOURCE_EMPTY_EVIDENCE_LINES 21`、`MOMO_IMPORT_CONFIG 當日業績匯入|即時業績_當日`、`MOMO_LATEST_IMPORT_JOB 56|completed|即時業績_當日.xlsx|2026-06-18T11:41:00.853176|2026-06-18T11:42:02.309425|10936|10936|0` | 2026-06-24 23:04 repo-side cold-start v1.42 dry-run returns `PASS=88 WARN=0 BLOCKED=1` and classifies the only blocker as: ```text 188 momo source file absent while daily sales data stale ``` This is repo-side source-of-truth enhancement only. 2026-06-24 23:15 read-only deploy parity check proves the live 110 script is still older: repo hash `f60b81029969a527dc742ebc9558d2933f11fe24ec4f46f7a7bc6637759b7b05`, live hash `10608873d406911a519afa96218abebc2b85ab6123bdf46b6e21eb269e554bb8`. Do not claim the live 110 deployed script has this v1.42 behavior until `/home/wooo/scripts/full-stack-cold-start-check.sh` is synced under an approved change and its hash/readback is recorded through §13.3.1. GO / NO-GO: ```text GO: declare MOMO web/API/container/database service available. GO: declare current-month table parity good. NO-GO: declare MOMO business data current. NO-GO: declare FULL_STACK_GREEN while MOMO_DAILY_FRESHNESS > 2. NO-GO: re-import old archived files to fake freshness. NO-GO: import product exports or manually constructed spreadsheets as daily sales source. NO-GO: truncate tables, restore whole DB, or move Drive files when sync_success is false. ``` 解除 blocker 的唯一合格證據： ```text 1. New legitimate 即時業績_當日 source file appears in the expected Drive intake path, or owner supplies a verifiable source-evidence reference. 2. Import job completes with success=true and sync_success=true. 3. Drive file movement / archive evidence shows the source was handled once. 4. daily_sales_snapshot and realtime_sales_monthly counts and date bounds match for the imported range. 5. MOMO_DAILY_FRESHNESS <= 2. 6. backup / offsite / cold-start scorecard rerun after import remains green except known DR escrow blocker. ``` 如果 source file 缺席，正確回報是： ```text MOMO service is recovered, data pipeline is waiting for upstream source file. No safe import candidate exists. Full-stack remains blocked by data freshness, not by service outage. ``` ### 14.32 2026-06-24 188 nginx-exporter / CD monitoring coverage gate 2026-06-24 的第六段變更是把 CD post-deploy monitoring coverage 失敗納入重啟 SOP。`2ec7f6f4` 的 runtime deploy 已回寫 `622bc372` 並且 production API health 為 healthy，但 CD `#3294` 的 post-deploy checks 因 `nginx-exporter` target down 留下 Failure。根因是 188 `nginx-exporter` container 未運行，並非 Nginx public gateway、API/Web rollout 或 MOMO 服務故障。 | 項目 | 20:10 monitoring coverage baseline | |------|------------------------------------| | SOP version | `v1.34` | | Affected CD run | Gitea CD `#3294` 歷史結果仍為 Failure；deploy marker `622bc372` 已寫入 | | Root cause | Prometheus job `nginx-exporter` down，target `192.168.0.188:9113` connection refused | | Non-root cause | Nginx `stub_status` 正常；不需要 reload Nginx、不需要重啟 API/Web/MOMO、不需要改 firewall | | Live restore source | `/home/ollama/nginx-exporter.yml` | | Repo helper | `scripts/ops/188-nginx-exporter-restore.sh` | | Check mode | `--check` only reads stub_status, compose config, container state, and metrics | | Apply mode | `--apply` runs `docker compose -f /home/ollama/nginx-exporter.yml up -d` after stub_status and compose config pass | | Exporter metrics | `nginx_up 1`、`nginx_connections_active`、`nginx_http_requests_total` | | Monitoring coverage | `Jobs 總數=14`、`全部 UP=14`、`真實問題=0`、`預期覆蓋率=100.0%` | | Declaration limit | 可宣稱 exporter / monitoring coverage recovered；不可把歷史 CD run 改稱 success，也不可宣稱 full-stack green / DR complete | Post-reboot / post-CD 188 nginx-exporter check: ```bash bash scripts/ops/188-nginx-exporter-restore.sh --check python3 scripts/generate_monitoring.py --check --stabilization-sleep-seconds 0 ``` 如果 `--check` 只在 metrics 階段失敗，但 `stub_status` 與 compose config 都通過，且維護窗口允許恢復無狀態 exporter： ```bash bash scripts/ops/188-nginx-exporter-restore.sh --apply python3 scripts/generate_monitoring.py --check --stabilization-sleep-seconds 0 ``` 禁止把這個症狀用下列方式處理： ```text NO-GO: reload Nginx before stub_status / exporter metrics prove Nginx config is the issue. NO-GO: restart product containers because monitoring coverage alone is red. NO-GO: silence monitoring coverage or mark CD green without target recovery evidence. NO-GO: prune Docker volumes or delete exporter state not owned by this SOP. ``` ### 14.33 2026-06-24 MOMO V10.646 / source-file absence / dual-workstation baseline 2026-06-24 的第七段變更是把 MOMO 的「程式版本最新」與「業務資料不新」拆成兩個獨立 gate，並把 Mac Mini / MacBook Pro 的 MOMO Codex 工作區固定到 Gitea main 最新基準。這避免重啟後出現兩種誤判：看到 `/health` 最新版就宣稱資料已更新，或看到資料 stale 就誤以為服務仍是舊版。 | 項目 | 20:42 MOMO / workstation baseline | |------|-----------------------------------| | SOP version | `v1.35` | | MOMO public health | `https://mo.wooo.work/health` returns healthy, version `V10.646` | | Gitea main truth | `wooo/ewoooc` `main=7cfca9375445ea03d6f5d10512d0276a20914d25`, `SYSTEM_VERSION = "V10.646"` | | Mac Mini workspace | `/Users/ogt/codex-workspaces/momo-pro-dev`, branch `codex/momo-current-main-dev-base-20260624`, commit `7cfca9375445ea03d6f5d10512d0276a20914d25`, dirty `0` | | MacBook workspace | `/Users/ooo/codex-workspaces/momo-pro-dev`, branch `codex/momo-current-main-dev-base-20260624`, commit `7cfca9375445ea03d6f5d10512d0276a20914d25`, dirty `0` | | Remote baseline branch | `wooo/ewoooc` `codex/momo-current-main-dev-base-20260624` points to `7cfca9375445ea03d6f5d10512d0276a20914d25` | | DB parity | current-month `daily_sales_snapshot` and `realtime_sales_monthly` match at `10936` rows, range `2026-06-01..2026-06-17` | | Data freshness | `MOMO_DAILY_FRESHNESS 7|2026-06-17`; still a hard blocker | | Source candidates inspected | Mac Mini current daily file contains only `2025-07-01..2025-07-02`; iCloud full-month file contains only `2025-06-01..2025-06-30`; MacBook candidates are header-only or the same `2025-07-01..2025-07-02` file | | Declaration limit | 可宣稱 MOMO release current 與 Codex dual-workstation baseline ready；不可宣稱 MOMO data current 或 full-stack green | MOMO post-reboot 判定必須同時回答四個問題： ```text MOMO_RELEASE_CURRENT = yes/no MOMO_DB_PARITY = yes/no MOMO_DATA_FRESH = yes/no MOMO_SOURCE_AVAILABLE = yes/no ``` 解除 MOMO data freshness blocker 的唯一安全路徑： ```text 1. 新的合法 即時業績_當日 source file 出現在預期 Drive intake，或 owner 提供可驗證的 source-evidence reference。 2. 匯入 job 成功，且同步 realtime_sales_monthly 失敗時不得標 completed。 3. source file movement / archive evidence 證明該檔只處理一次。 4. daily_sales_snapshot 與 realtime_sales_monthly row count / date bounds 一致。 5. MOMO_DAILY_FRESHNESS <= 2。 ``` 禁止把以下情境當成解除 blocker： ```text NO-GO: 用舊 archive、iCloud 舊月檔、header-only 檔或測試檔重複匯入。 NO-GO: 把 V10.646 health 當成資料日期已到今天。 NO-GO: 把 current-month parity 當成 data freshness。 NO-GO: truncate 或 restore 整庫來製造新鮮度。 ``` ### 14.34 2026-06-24 MOMO import sync failure boundary hardening 2026-06-24 21:57 的第八段變更是把 MOMO 自動匯入的「partial success」風險納入重啟 SOP。2026-06-24 22:17 已補正式 release readback：同一修正已 fast-forward 到 MOMO `main`，Gitea Actions `cd.yaml #904` 成功，188 live source marker 已確認。`daily_sales_snapshot` 寫入成功不代表整體匯入成功；`realtime_sales_monthly` 同步失敗時，必須 fail job、保留來源檔，不得移動 Google Drive 檔案到 archive。 | 項目 | 22:17 MOMO import-boundary production baseline | |------|------------------------------------------------| | SOP version | `v1.40` | | Production health | `https://mo.wooo.work/health` healthy, version `V10.653` | | Live DB read-only | `daily_sales_snapshot=104614 rows, 2025/07/01..2026/06/17`; `realtime_sales_monthly=786621 rows, 2024/01/01..2026/06/17` | | Scheduler read-only | 最近 12 小時 `當日業績匯入` / `即時業績_當日` 均為 `0` 個 Excel，排程不發送成功通知 | | Latest successful import | job `56 completed`, `10936` rows, `2026-06-18 11:41..11:42` | | Code / deploy | MOMO `main` and `codex/momo-current-main-dev-base-20260624` commit `84035906aba0e5e190d031a13cfd9b47a8cd1f73`; Gitea Actions `cd.yaml #904` Success | | Live source marker | 188 `/home/ollama/momo-pro/services/import_service.py` contains `_table_columns`, `業績分析儀表板同步失敗`, and `保留來源檔案等待重試，不移動 Google Drive 檔案` | | Regression | `pytest tests/test_import_service_sql_params.py tests/test_auto_import_data_sync.py tests/test_auto_import_failure_boundaries.py -q` => `10 passed` | | Production deploy state | Production patched for code boundary; data freshness still blocked until a legitimate newer source file imports successfully | MOMO import success 判定： ```text GO: process_daily_sales_import returns True only if daily_sales_snapshot write and realtime_sales_monthly sync / verification both pass. GO: auto_import_from_drive may move the Drive source file only after process_daily_sales_import returns True. NO-GO: mark import_jobs.status=completed when sync_success=false. NO-GO: move or archive the Drive source file when realtime_sales_monthly sync failed. NO-GO: send a generic success notification for file_count > 0 before verify_import_data_sync passes. ``` 重啟後若 MOMO data freshness blocked，先分成三層，不要混在一起： ```text 1. Service availability: /health, container, DB connection. 2. Source availability: Drive pending folder has a legitimate new 即時業績_當日 source file. 3. Data correctness: import job completed with sync_success=true, and daily_sales_snapshot / realtime_sales_monthly match the imported date range. ``` ### 14.35 2026-06-25 MOMO preflight 與 110 CPU orphan Chrome 分流 2026-06-25 11:01 的第九段變更是把兩個常見誤判收斂成可重跑 SOP： 1. MOMO service health green 不等於 data fresh。 2. 110 high load 不等於可以重啟 Docker 或取消 CI。 MOMO 專用 preflight： ```bash scripts/reboot-recovery/momo-drive-token-source-recovery-preflight.sh ``` 此腳本只做 read-only SSH / Docker metadata / logs / DB query，不讀 token 內容、不 import、不移動 Drive 檔、不 restart。14:16 live result: ```text MOMO_DRIVE_TOKEN_SOURCE_PREFLIGHT PASS=18 WARN=3 BLOCKED=0 HOST=ollama@192.168.0.188 FRESHNESS_MAX_DAYS=2 MOMO_PUBLIC_HEALTH_CODE 200 MOMO_HEALTH_CODE 200 MOMO_HEALTH_VERSION V10.674 MOMO_APP_HEALTH healthy SCHEDULER_RUNNING true SCHEDULER_HEALTH healthy SCHEDULER_RESTART_COUNT 0 TELEGRAM_BOT_HEALTH healthy MOMO_CONTAINER_REPLACE_EVENTS_45M 11 TOKEN_STAT 100000:100000:600 CONTAINER_TOKEN_STAT 0:0:600 LOCAL_EXACT_DAILY_SOURCE_COUNT 0 LOCAL_EXACT_DAILY_SOURCE_LATEST none DB_DAILY 109061|2025-07-01|2026-06-24 DB_MONTHLY_SYNC 15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24 DB_DAILY_FRESHNESS 1|2026-06-24 DB_LATEST_DAILY_IMPORT_JOB 57|completed|即時業績_當日.xlsx|2026-06-25T13:16:47.359958|2026-06-25T13:18:02.964985|15383|15383|0 ``` 110 CPU 分流： | Evidence | Decision | |----------|----------| | `ps` shows `stockplatform-review-bulk-ux` Chrome groups with root process PPID `1`, no parent node smoke, and sustained high CPU | Treat as orphan browser smoke. Run dry-run if available, then only with owner approval use targeted `SIGTERM` by process group. | | Active Gitea Actions container is consuming CPU, e.g. `GITEA-ACTIONS-TASK-*`, `next build`, `uv pip install`, `docker-buildx` | Treat as legitimate CI/CD load. Do not kill unless there is explicit release owner approval to cancel the run. | | `vmstat` shows high iowait or active swap in/out | Treat as storage / memory pressure, not browser runaway. Do not kill random processes; capture disk / memory evidence first. | 2026-06-25 10:58 user-approved action: ```text Targeted command type: process SIGTERM only. Targeted process groups: 438005, 471295, 640155, 670628. Scope: orphan `stockplatform-review-bulk-ux` Chrome groups on 110. Post-check: `OLD_GROUPS_REMAINING` empty. Not performed: Docker restart, systemd restart, Nginx reload, firewall/iptables change, K8s action, CI cancellation, Wazuh/SOC change, secret read. Remaining load: active Gitea Actions / CI build work; observe queue and timeout instead of killing. ``` ### 14.22 重啟後時間軸驗證 每次重啟後照時間軸推進，不要等到最後才一次判定。 | 時間點 | 目標 | 必跑證據 | 可以宣稱 | |--------|------|----------|----------| | `T+0` | power / VM / console 已開始 | console / hypervisor / UPS / operator note | maintenance started | | `T+5m` | LAN / SSH 回復 | ping、ARP、SSH port、`who -b` | `HOST_BOOTED` | | `T+15m` | 主機基礎服務回復 | `systemctl is-system-running`、failed units、Docker / PostgreSQL / Redis / K3s role checks | `HOST_READY` | | `T+30m` | 核心服務回復 | 188 DB、110 Harbor/Gitea/Prom/AM、K3s nodes、AWOOOI API/Web、public routes | `SERVICE_READY` for scoped hosts | | `T+45m` | 排程與資料一致性 | backup status、offsite verifier、momo DB parity、CronJobs、alert visibility | service recovery confidence | | `T+60m` | 釋出高負載與自動化 | cold-start scorecard、load/core、runner guardrails、AI observe-only gate | release runner/CD only if gates allow | 若任一時間點卡住，記錄卡在哪個 gate，不要跳到下一層。連續兩次重啟都卡同一 gate，必須回寫 §16 Known Drift 或 workplan。 --- ## 15. Done Criteria All must be true: - Four hosts reachable by SSH. - 188 PostgreSQL and Redis healthy. - 110 Harbor, Gitea, Prometheus, Alertmanager healthy. - 120/121 K3s nodes Ready. - VIP `192.168.0.125` present. - AWOOOI API and Web reachable through NodePort/VIP. - Alertmanager E2E webhook succeeds. - cron/CronJob schedules are active, unsuspended, and verified. - MOMO release version matches Gitea source-of-truth for the intended deployment branch. - momo `daily_sales_snapshot` 與 `realtime_sales_monthly` 在最新匯入日期範圍內筆數一致。 - momo business data freshness is within the declared SLO, and the latest import source evidence is legitimate; DB parity alone is not enough. - Sentry and SignOz are either healthy or explicitly in controlled backlog recovery. - High-load batch services are capped or delayed. - Runners are guarded and released last. - AI auto-remediation is not in full execution mode until all gates are green. - 110 cold-start textfile monitor is fresh, and Prometheus has the cold-start alert rules loaded. - 110 runaway process textfile monitor is fresh, and Prometheus has `HostOrphanBrowserSmokeHighCpu` plus CI load classification rules loaded. - 110 global `/home/wooo/.ssh/known_hosts` still contains verified 120 / 188 entries after any CD run; deploy jobs use `/home/wooo/.ssh/deploy_known_hosts` only. ### 15.1 可宣稱狀態 | 可宣稱文字 | 必要條件 | |------------|----------| | `110 host recovered` | 110 `HOST_READY`，failed units `0` 或全部可解釋，核心端口與 cron / backup status 已查 | | `public core services recovered` | public routes/TLS 2xx/3xx，AWOOOI API health、Harbor/Gitea/Stock/Sentry/SignOz/Langfuse/Bitan smoke OK | | `backup/offsite current` | `backup-status --no-notify` 無 stale，offsite verifier `VERIFY_OK=1`，且任何 failed component 有明確 owner | | `service recovery with known blocker` | cold-start `BLOCKED` 只剩已知 blocker，例如 120；告警保持可見 | | `full-stack green` | §15 全部成立，cold-start `WARN=0 BLOCKED=0` | | `DR complete` | full-stack green 且 credential escrow missing count 為 0 | --- ## 16. Known Drift To Fix After Recovery 這些項目必須在事故後整理，不要在 P0 恢復中途順手大改。 - `SERVICE-ENDPOINTS.md` still has old Prometheus/Alertmanager locations. - Audit older docs for direct node webhook targets; current main path should be VIP `192.168.0.125:32334`. - OpenClaw `8088` vs `8089` must be live-confirmed and normalized. - 188 compose paths drift between `/home/ollama/*` and Ansible `/opt/*`. - 110 runner docs still mention Docker runner in places; live startup prefers host `gitea-act-runner-host.service`. - `scripts/setup-runner-watchdog.sh` conflicts with the 2026-05-05 runner watchdog disablement guardrail. - `grist.wooo.work` / `registry.wooo.work` public HTTP/HTTPS currently route to `aiops.wooo.work`; their old 110 certbot renewal configs are disabled until public routing is corrected or DNS-01 renewal is configured. - `stockplatform-shared-ui-monitor.timer` / service source-of-truth 仍需清理或重建；2026-06-12 只停用 stale timer 以解除 host degraded。 - 111 local Ollama fallback 目前不可達；production provider 由 GCP-A / GCP-B 承接，但 111 恢復應另列 AI provider resilience 工作。 - 本 SOP v1.5 新增內容已用繁體中文補強；舊章節仍有英文段落，後續 runbook hygiene 應分批翻譯，不要在事故 P0 中混入大規模格式重排。 ### 16.1 188 startup / ClawBot rebuild 不得阻塞 SLO 2026-06-30 主機重啟後，188 `awoooi-startup.service` 在 `TimeoutStartSec=300` 內被 systemd 砍掉；live journal 顯示卡在 ClawBot `docker compose build --no-cache`，但事後 `clawbot` / `clawbot-redis` 容器已 healthy。此類情況不可再解讀為需要盲目 restart / rebuild；正確做法是讓 startup oneshot 完成，將 ClawBot 健康交給 post-reboot verifier 判斷。 新的 source-of-truth： - `scripts/reboot-recovery/awoooi-startup.sh` - `scripts/reboot-recovery/awoooi-startup.service` - `scripts/reboot-recovery/tests/test_188_host_hygiene_checklist.py` 規則： - ClawBot startup 只允許 bounded `docker compose up -d`，預設確認窗口 `CLAWBOT_UP_TIMEOUT_SECONDS=90`。 - ClawBot image rebuild 預設停用：`CLAWBOT_STARTUP_REBUILD_ALLOWED=0`。 - 只有在 maintenance / controlled recovery 明確設定 `CLAWBOT_STARTUP_REBUILD_ALLOWED=1` 時，才允許 bounded rebuild，且 `CLAWBOT_BUILD_TIMEOUT_SECONDS=120`。 - 不得在 reboot startup path 預設執行 `docker compose build --no-cache`。 - `awoooi-startup.service` timeout 對齊 10 分鐘 SLO：`TimeoutStartSec=600`。 - 若 ClawBot 未確認 running，startup script 只留下 warning；`post-reboot-readiness-summary.sh` / `full-stack-cold-start-check.sh` 必須負責把它標成服務 blocker。 驗證命令： ```bash bash -n scripts/reboot-recovery/awoooi-startup.sh DATABASE_URL=sqlite+aiosqlite:////tmp/awoooi-codex-api-test.db PYTHONPATH=apps/api \ python3.11 -m pytest scripts/reboot-recovery/tests/test_188_host_hygiene_checklist.py -q ``` ### 16.2 188 CD/API test 與 110 systemd exporter 壓力不得混成一般 CPU 高 2026-07-01 post-reboot pressure 回讀補充兩個獨立模式： 1. 188 `k3s-postgres-recovery` 高 CPU 可能不是 DB service 壞掉，而是 CD / backup / export 正在讀 production DB。 2. 110 `systemd-logind` / DBus 卡住時，`systemd_units.prom` exporter 反覆 `systemctl show` runner units 會放大控制面壓力，讓 SSH / `systemctl list-*` timeout。 188 判讀順序： | Evidence | Decision | |----------|----------| | `pg_stat_activity.application_name=pg_dump` 且 query 為 `COPY ... TO stdout` | 視為備份 / 匯出壓力；先定位 host process group，只允許停止該讀取型 export，不重啟 PostgreSQL / Docker。 | | `docker ps` 出現 `awoooi-cd-*-api-tests` 且 `k3s-postgres-recovery` 同步高 CPU | 視為 CD API test 壓力；先 drain non-prod runner / test container，再把 CD gate 補上 188 Docker metrics。 | | `docker_stats.prom` 中 `docker_container_cpu_cores{container_name="k3s-postgres-recovery"}` 超過 `2.0` | `scripts/ci/wait-host-web-build-pressure.sh` 必須 fail-fast，不得啟動新 API tests。 | 110 判讀順序： | Evidence | Decision | |----------|----------| | journal 出現 `systemd-logind` pending replies exhausted / `systemctl list-*` timeout / `sshd MaxStartups throttling` | 視為 control-plane saturation；停止並行 SSH，改成單線控制或 Prometheus readback。 | | `systemd_units.prom` 仍出現多個 `systemctl show ... timed out after 10 seconds` label | 視為舊 exporter 放大 DBus 壓力；套用新版 exporter 或暫停舊 exporter，不要先重啟主機。 | | `gitea` 固定接近 CPU cap 且 no active Actions containers | 視為 Gitea queue / hook backlog；不要恢復 generic runner，不要把 host pressure guard 改 warn-only。 | 新的 source-of-truth： - `scripts/ci/wait-host-web-build-pressure.sh` - `scripts/ops/systemd-units-textfile-exporter.py` - `scripts/ops/tests/test_systemd_units_textfile_exporter.py` - `docs/operations/host-cpu-pressure-drain-readback-2026-07-01.snapshot.json` 驗證命令： ```bash bash -n scripts/ci/wait-host-web-build-pressure.sh scripts/ops/systemd-units-textfile-exporter.py python3.11 -m pytest scripts/ops/tests/test_systemd_units_textfile_exporter.py ops/runner/test_cd_controlled_runtime_profile.py -q ``` ### 16.3 2026-07-01 post-reboot resource / disk / detector closure 2026-07-01 全主機重啟後，runtime 問題分成四條：188 `momo-db` 缺 Docker resource limit、110 Sentry relay memory-limit pressure、110 / 188 root filesystem 超過 disk-pressure 門檻、以及 reboot detector scope 仍停在舊 `110_120_121_188`。這些不是噪音；disk pressure 會阻塞 Docker / registry / build path，resource-limit pressure 會讓告警長期失真，detector scope 錯誤會讓「有沒有偵測到重啟」本身不可信。 已完成 live 修復： | Layer | 修復 | 讀回證據 | |-------|------|----------| | 188 `momo-db` guardrail | `docker update --cpus=2 --memory=4g --memory-swap=6g momo-db` | `nanocpus=2000000000 memory=4294967296 memswap=6442450944 restart=unless-stopped` | | 110 Sentry relay guardrail | `docker update --memory=3g --memory-swap=4g sentry-self-hosted-relay-1` | `health=healthy memory=3221225472 memswap=4294967296`，`https://sentry.wooo.work/_health/` 回 `ok` | | 110 disk pressure | 只清 unreferenced dangling images 與 bounded builder cache；不使用 `docker system prune` | `/` 從約 `85%` 降到 Prometheus `79.6829%` | | 188 disk pressure | builder-cache-only follow-up；不碰 containers / volumes / DB / backups | `/` 從約 `85%` 降到 Prometheus `79.2250%` | | reboot detector | 安裝新版 auto-recovery SLO textfile scope | `awoooi_reboot_event_detected{scope="99_110_111_112_120_121_188"} 1` | | full-stack scorecard | 重新跑全主機 cold-start | `PASS=96 WARN=0 BLOCKED=0`，result `GREEN` | 新的 source-of-truth： - `scripts/ops/docker-disk-pressure-retention-cleanup.py` - `scripts/ops/tests/test_docker_disk_pressure_retention_cleanup.py` - `ops/reboot-recovery/full-stack-cold-start-baseline.yml` - `docs/operations/post-reboot-runtime-recovery-readback-2026-07-01.snapshot.json` Docker disk pressure cleanup 規則： - 預設 dry-run；只有明確 `--apply` 才執行。 - 不使用 `docker system prune`。 - 不刪除 containers、volumes、databases、backups、logs。 - dangling image cleanup 只允許移除未被任何 container 參照、沒有 tag、超過 age gate 的 image，且保留最新 N 個。 - BuildKit cache cleanup 必須明確加 `--include-builder-cache`，並設定 `--builder-keep-storage`。 - `--min-age-hours=0` 只允許和 `--skip-dangling-images` 一起使用，避免無 age gate 刪 image。 標準命令： ```bash python3 scripts/ops/docker-disk-pressure-retention-cleanup.py --host-label 110 --include-builder-cache python3 scripts/ops/docker-disk-pressure-retention-cleanup.py --host-label 110 --apply --min-age-hours 24 --keep-dangling-newest 20 --include-builder-cache --builder-keep-storage 30GB python3 scripts/ops/docker-disk-pressure-retention-cleanup.py --host-label 188 --skip-dangling-images --apply --min-age-hours 0 --include-builder-cache --builder-keep-storage 1GB ``` 宣告限制： - 可以宣稱：這次回讀的 runtime full-stack 已 GREEN，110 / 188 disk pressure、momo-db missing resource limit、Sentry relay memory pressure、RebootEventDetectorMissing 已解除。 - 不可宣稱：這次重啟有達成 10 分鐘 SLO。Prometheus 仍正確保留 `RebootAutoRecoverySLOMissed`，代表本次事件已 missed，必須做為後續自動化改善 evidence，而不是消音。 - 不可用 stale API snapshot 覆蓋 live truth；`https://aiops.wooo.work/api/v1/agents/reboot-auto-recovery-slo-scorecard` 若 `generated_at` 舊於最新 textfile / Prometheus readback，只能列為 stale artifact。 ### 16.4 2026-07-02 110 StockPlatform DB pressure guardrail 2026-07-02 00:20 後，`Host110SustainedModeratePressure` 正確觸發。這不是缺監控；Prometheus 已主動抓到 110 壓力。即時 evidence 顯示 orphan browser group `0`、active Gitea Actions process `0`、swap ratio 約 `0.0127`，壓力來源是 `gitea` 接近既有 2 CPU cap，以及 `stockplatform-v2-postgres-1` 沒有 Docker resource guardrail、曾瞬間到約 `292.96% CPU`。 已完成 live 修復： ```bash docker update --cpus=1.5 --memory=4g --memory-swap=6g --restart unless-stopped stockplatform-v2-postgres-1 ``` 讀回證據： | Check | Evidence | |-------|----------| | Docker inspect | `running=true health=healthy nanocpus=1500000000 memory=4294967296 memswap=6442450944 restart=unless-stopped` | | Direct Docker stats | `stockplatform-v2-postgres-1 cpu=149.38% mem=506.2MiB / 4GiB` after apply | | Prometheus convergence | `stockplatform-v2-postgres-1=1.6508 cores`、`gitea=1.8097 cores`、`load5/core=0.673333` | | Product route | `https://stock.wooo.work/`、`/healthz`、`/api/healthz` all `200` | | Product data freshness | `/api/v1/system/freshness` `status=ok`、`latest_trading_date=2026-07-01` | | Alert readback | `Host110SustainedModeratePressure` cleared；firing alerts only `HostRebootEventDetected` and `RebootAutoRecoverySLOMissed` | 後續同一輪 Gitea 仍曾貼近 2 CPU cap 並觸發 `DockerContainerCpuSustainedHigh`；已再套用： ```bash docker update --cpus=1.5 gitea ``` 讀回：`nanocpus=1500000000 memory=3221225472 restart=always`、Gitea API `/api/v1/version` 回 `1.25.5`。Prometheus 收斂後 `load5/core=0.725833`、`gitea=0.9882 cores`、`stockplatform-v2-postgres-1=0.7296 cores`，CPU 類 firing alerts 清空。 宣告限制： - 可宣稱：110 sustained moderate CPU alert 已主動觸發、已定位、已用 Docker guardrail 降壓，StockPlatform route / health / freshness 未受損。 - 不可宣稱：110 永久不會再高負載；Gitea queue / hook backlog 仍需在後續 queue / control-path lane 持續觀察。 - 不得用 DB restart、container restart、manual DB write、query kill 或 fake freshness 來處理 StockPlatform pressure。 ### 16.5 2026-07-02 hostedtoolcache / pnpm install 必須算 active CI load 同一輪 110 high load 診斷發現 `node /opt/hostedtoolcache/node/.../pnpm install --frozen-lockfile` 會造成短期 runnable burst。這是 Gitea Actions 類 CI install load，但舊 exporter 只認 `/.cache/act/`、Docker build、BuildKit、`next build`，導致 `awoooi_host_gitea_actions_active_process_count` 誤報 `0`，進而讓 controller 看起來像 unknown / Gitea backlog，而不是 active CI。 修正 source-of-truth： - `scripts/ops/host-runaway-process-exporter.py` - `scripts/ops/host-sustained-load-evidence.py` - `scripts/ops/tests/test_host_runaway_process_exporter.py` 新增分類規則： - `/opt/hostedtoolcache/` - `pnpm install` - `npm ci` - `yarn install` 宣告限制： - active CI load 只能走 runner / queue / pressure gate lane；不得直接 `SIGKILL`、`docker restart`、`systemctl restart`、DB restart 或恢復 generic runner。 - evidence output 仍必須保持 sanitized，不輸出 hostedtoolcache 完整路徑、raw command、URL、workspace path 或 secret。 ### 16.6 2026-07-03 backup alert receipt 四段證據 P0-5 / P0-6 不得再只用「backup health exporter 有跑」或「Prometheus rule 存在」宣稱備份告警閉環。`scripts/ops/backup-health-textfile-exporter.py` 必須輸出 backup alert receipt requirement，且每個 required backup alert 都要追蹤四段不含 secret 的 evidence marker： - `sent` - `received` - `dedup` - `escalation` 標準 marker 位置： ```bash /backup/alert-receipts/..last_success /backup/alert-receipts//.last_success /backup/alert-receipts/.last_success /backup/alert-receipts//last_success ``` shared marker 只作兼容讀回；正式 closeout 仍應逐 stage 補齊。marker 內容只允許 timestamp 或脫敏 evidence id，不得寫入 Telegram token、chat raw payload、使用者訊息、runner token、DB DSN、`.env`、cookie、authorization header 或任何 secret。 新增 metrics： ```text awoooi_backup_alert_receipt_expected_info{host,required_alert,scope,required_notification_type,receipt_channel,max_age_hours} awoooi_backup_alert_receipt_fresh{...} awoooi_backup_alert_receipt_stage_expected_info{...,stage} awoooi_backup_alert_receipt_stage_fresh{...,stage} ``` Prometheus 必須載入： - `BackupAlertReceiptMetricMissing` - `BackupAlertReceiptMetricMissing188` - `BackupAlertReceiptStageMissing` 判讀規則： - `awoooi_backup_alert_receipt_stage_fresh == 0` 代表該 backup alert 的 `sent / received / dedup / escalation` 某段沒有新鮮證據；不能宣稱 Telegram/AwoooP receipt 完成。 - `BackupAlertReceiptStageMissing` 必須聚合成每個 `host / receipt_channel` 一條 alert；明細用 `awoooi_backup_alert_receipt_stage_fresh{host="...",receipt_channel="..."} == 0` 查 `required_alert / scope / stage / required_notification_type`，避免每個 stage 都打 Telegram 造成噪音。 - `required_alert` 是原始 backup alert 名稱；不得用 Prometheus 保留 label `alertname` 承載原始 alert，避免被 meta alert 覆蓋。 - `required_notification_type` 是原始 backup alert 應走的類型；meta alert 自己的 `notification_type` 可維持 `TYPE-1`。 - `scripts/ops/backup-alert-label-contract-check.py` 是 source rule contract；`scripts/ops/backup-alert-live-visibility-check.py` 是 Prometheus / Alertmanager live visibility contract。 禁止事項： - 不得為了產生 receipt 而送測試 Telegram、呼叫 Bot API、改 Alertmanager receiver route、改 silence 或 workflow_dispatch。 - 不得讀取 token / chat raw payload / secret volume。 - 不得用 receipt marker 取代真正 backup freshness、restore drill、Gitea bundle completeness 或 offsite verifier。