diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index df364d3aa..3d81071ec 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,21 @@ +## 2026-07-03 — 03:55 P0-006 production scorecard 收斂與 SOP 沉澱 + +**完成內容**: +- 接續 2026-06-30 全主機重啟事故主線,將 reboot SLO scorecard 的 service/data/backup/Wazuh/Gitea bundle false-red 收斂完成;production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 03:55 readback 回 `active_blocker_count=8`、`readiness_percent=67`、`can_claim_all_services_recovered_within_target=false`。 +- 已確認可宣稱的綠燈:`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`HOST_188_SERVICE_GREEN=1`、`WAZUH_DASHBOARD_DEGRADED=false`、Gitea repo bundle backup `expected_repo_count=12` / `repo_row_count=12` / `all_expected_ok=true`、StockPlatform freshness `status=ok` / `latest_trading_date=2026-07-02` / blockers `[]`。 +- 剩餘真 blocker 固定為 host boot detection 與 Windows99 VMware lane:111 unreachable、99 uptime unknown、fresh all-host reboot event missing、Windows99 VMX missing alias `111`、guest power not ready `111/112/120/121/188`;Windows Update no-auto-reboot readback 為 ready,但 Windows99 no-secret remote execution channel 仍 blocked。 +- `FULL-STACK-COLD-START-SOP.md` 升到 v1.102:固定 exporter timeout 不得抹掉 post-start 綠燈摘要,並把 deploy marker 切換時 public API/SLO route 502 且無 maintenance fallback body/header 定為 P0 public UX blocker。 + +**驗證**: +- Production SLO API:`generated_at=2026-07-03T03:55:34+08:00`,active blockers 為 `all_required_hosts_not_in_10_minute_reboot_window`、`fresh_all_host_reboot_event_missing`、`host_boot_observation_older_than_target_window`、`host_unreachable_after_reboot`、`host_uptime_unknown`、`reboot_event_required_host_unreachable`、`windows99_vmware_guest_power_not_ready`、`windows99_vmware_vmx_missing`。 +- Production health API 回 HTTP 200,但整體 `status=degraded`,原因包含 SignOz connection refused 與 local Ollama cooldown;這不是 10 分鐘 SLO 完成證據。 +- StockPlatform `/api/v1/system/freshness` 回 HTTP 200、`status=ok`、`latest_trading_date=2026-07-02`,核心價量、三大法人、融資融券與 AI recommendations 均為 `ok`。 +- Gitea SSH `refs/heads/main` 回 `89d4d611262bfec2fdbfbf8709243ba316c1c91b`,對齊 deploy marker `89d4d6112 chore(cd): deploy 17ba08c [skip ci]`。 + +**仍維持**: +- 不能宣稱所有主機重啟後 10 分鐘內自動恢復;不能宣稱 99 VMware guest autostart 全部 ready;不能把 deploy cutover 502 當成正常噪音。 +- 未讀 secret / token / `.env` / raw sessions / SQLite / auth;未使用 GitHub / gh;未 workflow_dispatch;未重啟 host / VM / service;未 Docker / Nginx / K3s / DB / firewall restart;未 DROP / TRUNCATE / restore / prune / delete / force push。 + ## 2026-07-03 — 02:40 Knowledge Base 分類矩陣與 RAG readback 修正 **完成內容**: diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index b42c019de..d645ff1cf 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,8 +1,8 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.100 +> Version: v1.102 > Last updated: 2026-07-03 Asia/Taipei -> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path. +> Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號,但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal;此納入不代表授權任何破壞性 runtime apply。 --- @@ -34,6 +34,10 @@ v1.99 runtime blocker action-matrix classification rule:Prometheus / runtime o v1.100 host-probe bounded TCP verifier rule:`scripts/reboot-recovery/reboot-auto-recovery-host-probe.sh` 的 reachable-only TCP fallback 不得只依賴 `nc -w`;每一個 `nc -z` port probe 必須再包一層 `run_with_timeout "${TCP_CONNECT_TIMEOUT_SECONDS:-2}"`,避免單台主機或單個 port 卡住後讓後續主機缺列,造成 10 分鐘 SLO readback 無法判斷。若 verifier timeout 或主機缺列,`reboot-event-detector.py` / `reboot-auto-recovery-slo-scorecard.py` 必須 fail-closed 成 host boot detection blocker;不得用前半段 partial probe 宣稱 all-host observed、fresh reboot 或 auto-recovery ready。2026-07-03 live no-write 驗證 artifact `/tmp/awoooi-reboot-host-probe-confirm-20260703-015844`:bounded host probe 約 `10.825s` 完成、`HOST_BOOT` rows `7`、`missing_hosts=[]`;但 111 仍 unreachable、99 仍 `uptime_seconds=unknown`、188 仍 `systemd_state=degraded` / `startup_active=failed`,因此 10 分鐘全主機自動恢復 SLO 仍維持 blocked。 +v1.101 reboot SLO exporter timeout fallback rule:`scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh` 不得在 `post-reboot-readiness-summary.sh` timeout 時只寫 `POST_REBOOT_READINESS_SUMMARY_TIMEOUT=1` 而抹掉已完成的 `post-start-quick-check.log` 綠燈證據。timeout 預設為 `POST_REBOOT_READINESS_TIMEOUT_SECONDS=120`;若 full summary timeout 但 post-start log 已有摘要,exporter 必須寫出 partial `summary.txt` 並包含 `POST_REBOOT_READINESS_PARTIAL_FROM_POST_START=1`、`SERVICE_GREEN`、`PRODUCT_DATA_GREEN`、`BACKUP_CORE_GREEN`、`POST_START_RESULT`、`POST_START_BLOCKED` 等欄位。scorecard 不得只因 timeout artifact 就把 service / product data / backup core 重開成 active blocker;尚未讀回的 188 hygiene、Wazuh、DR 或 Windows99 欄位必須標成 partial / unknown,由各自 blocker lane 處理。 + +v1.102 production scorecard / deploy cutover rule:2026-07-03 03:55 production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 的目前事實是 `active_blocker_count=8`、`readiness_percent=67`、`can_claim_all_services_recovered_within_target=false`;`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`HOST_188_SERVICE_GREEN=1`、`WAZUH_DASHBOARD_DEGRADED=false`、Gitea bundle backup `expected_repo_count=12` / `repo_row_count=12` / `all_expected_ok=true`,StockPlatform freshness `status=ok` 且 `latest_trading_date=2026-07-02`。因此 service/data/backup/Wazuh/Gitea bundle blocker 不得因舊 Prometheus label 或 summary timeout 被重開;剩餘 P0 blocker 固定為 host boot detection / all-host fresh reboot event、111 unreachable、99 uptime unknown、Windows99 VMware VMX / guest power verifier。另,deploy marker 切換期間若 public API / SLO route 出現 502 且沒有 maintenance body / fallback header,即使稍後自癒,也必須列入 P0 public UX blocker;下一步固定是 source-controlled deploy drain、maintenance fallback verifier 與 public route watch,不得把短暫 502 當作正常噪音。 + 2026-07-02 110 control-path / Harbor recovery receipt rule:若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure,但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata,不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier,並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示:public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`;active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure,以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker,non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker;non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。 2026-07-02 110 controlled CD lane fail-closed enforcer staging rule:110 runner 壓力事故後,legacy / generic runner 仍必須 fail-closed;但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail,且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit,並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service;若 registration 缺失,readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail,不要手工反覆補同一組 artifact。 diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index 95dd2f340..8e00ccd5c 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -13,6 +13,21 @@ 本段覆蓋舊的「單次重啟後人工排查」做法。所有後續狀態回報必須依此順序推進;噪音若會遮蔽 P0,就掛回同一列,不另開支線。 +### 2026-07-03 03:55 最新 P0 覆蓋排序 + +下表覆蓋 2026-06-30 初始事故列;舊表保留為歷史追蹤。所有新插入需求必須掛在本表,不得再分散成臨時支線。 + +| 優先 | 狀態 | 工作項 | 最新證據 | 下一步 / 完成條件 | +|------|------|--------|----------|-------------------| +| P0-1 | BLOCKED_HOST_WINDOWS | 全主機 reboot auto-detection / auto-trigger / 10 分鐘恢復 SLO | 2026-07-03 03:55 production scorecard:`active_blocker_count=8`、`readiness_percent=67`、`can_claim_all_services_recovered_within_target=false`;`observed_host_count=7`、`missing_host_count=0`、`unreachable_host_count=1`、111 unreachable、99 uptime unknown。 | 先收斂 99 / Windows99 / VMware 與 111:恢復 no-secret management channel 或 console verify stdout,讀回 VMX / VM power / host uptime,再 rerun host probe + reboot-event detector;不得 reboot、不得 VM power change、不得讀 Windows 密碼。 | +| P0-2 | BLOCKED_PUBLIC_UX | Deploy / reboot 期間 public 502 維護頁與外部 fallback | 多次 deploy marker 切換時 SLO/API route 曾短暫 HTTP 502 且沒有 maintenance body / fallback header;目前 03:55 public maintenance runtime readback 為 ready、raw 5xx count `0`,但 cutover-time 502 未被 drain / watch 消除。 | 實作 source-controlled deploy drain / maintenance fallback verifier / public route watch;完成條件是 marker-time probe 不再看到 raw 502,或看到明確 maintenance fallback header/body。 | +| P0-3 | PARTIAL_GREEN_SOURCE_RUNTIME | 所有產品 / 網站版本與資料最新性 | Gitea `main=89d4d6112`;Production SLO readback 對齊 deploy marker `89d4d6112 chore(cd): deploy 17ba08c [skip ci]`;Stock freshness `status=ok`、`latest_trading_date=2026-07-02`、blockers `[]`。AWOOOI health HTTP 200 但整體 `degraded`,SignOz / local Ollama 仍需列為 runtime degraded evidence。 | 將 source SHA / deploy marker / runtime endpoint / public route watch 固定進 scorecard;完成條件是每個 public product 都有 source、deploy、runtime、freshness 四層 readback,且 degraded components 有 owner lane。 | +| P0-4 | BLOCKED_WINDOWS99_AUTOSTART | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | Scorecard:`windows99_update_no_auto_reboot_ready=true`、`windows99_vmware_verify_ready=false`、VMX missing alias `111`、powered off aliases `111/112/120/121/188`、WinRM unavailable、SSH BatchMode permission denied、RDP / Hyper-V console channel reachable。 | 只用 no-secret console / management collector 取得 `windows99-vmware-autostart.ps1 -Mode Verify` 輸出;完成條件是 VMX config ready、guest power ready、99 uptime known、all required host reachable。 | +| P0-5 | PARTIAL_GREEN_BACKUP_MONITORING | Gitea / 主機 / DB / 網站 / 服務 / 套件 / 工具 / log 備份監控告警 | Gitea repo bundle readback ready:expected `12`、rows `12`、missing `0`、failed `0`、sample restore dry-run ok;backup core green。這只證明 repo bundle / core backup,不等於 Gitea full dump、DB/settings/issues/packages/LFS、所有工具與 log 全量備份監控完成。 | 補齊 backup health textfile / Prometheus / Telegram receipt matrix:每個 backup scope 必須有 fresh、failed、age、restore-drill、alert-receipt;完成條件是沒有「不知道備份有沒有跑」的盲點。 | +| P0-6 | SOURCE_READY_ALERT_RECEIPT_BLOCKED | 主機關機 / 重啟 / SLO miss / backup failure Telegram 告警 | Source 已有 reboot / backup alert rules 與 per-blocker projection;scorecard `telegram_active_blocker_alert_required_count=8`。尚未完成 shutdown / reboot / backup alert 的脫敏 Telegram delivery receipt 全矩陣。 | 補 alert receipt readback:host down、host up、SLO miss、Windows99 blocker、backup stale/failed、deploy 502、freshness stale;完成條件是每類告警都有 sent / received / dedup / escalation evidence。 | +| P0-7 | SOURCE_READY_SLA_AUTOMATION | 固定排查順序、ETA / wait reason、自動化判斷與修復 | Scorecard 已固定 `current_phase=host_boot_detection_blocked`、`eta_or_wait_reason=reboot_event_readback_missing_eta_unavailable`、`fixed_triage_order`;但 10 分鐘內自動恢復仍未達標。 | 把每個 blocker 的 next_safe_action、post_verifier、forbidden_actions 接到自動 work item / Telegram / scorecard;完成條件是重啟後自動判斷、主動告警、主動 rerun verifier,不再人工臨場猜流程。 | +| P0-8 | PARTIAL_READY_POLICY | Windows99 禁止 Windows Update 無預警重啟 | Scorecard:`windows99_update_no_auto_reboot_ready=true`;但 Windows99 no-secret remote execution channel blocked,policy 證據仍需納入持續監控。 | 保留 readback,補週期性 verifier 與 Telegram drift alert;完成條件是 Windows Update policy drift 會自動告警且不需讀 secret。 | + | 優先 | 狀態 | 工作項 | 2026-06-30 證據 | 下一步 / 完成條件 | |------|------|--------|------------------|-------------------| | P0-1 | BLOCKED | 全主機 cold-start / 10 分鐘自動恢復 SLO | 23:27 live cold-start artifact `/tmp/awoooi-cold-start-after-3de828f97.log` 回 `PASS=67 WARN=5 BLOCKED=4`、`Result: BLOCKED`;blockers 是 110 registry external `/v2`、110 SSH read-only check、K3s registry pull refused by `110:5000`、SignOz TLS / public route。22:31 SLO scorecard `/tmp/awoooi-reboot-slo-live-20260630-2231-scorecard.json` 仍回 `can_claim_all_services_recovered_within_target=false`;22:28 post-reboot summary `/tmp/awoooi-post-reboot-readiness-20260630-222856/summary.txt` 回 `SERVICE_GREEN=0`、`PRODUCT_DATA_GREEN=0`、`BACKUP_CORE_GREEN=0`、`HOST_188_SERVICE_GREEN=0`。 | 先修第一個 runtime blocker:110 control path / Harbor registry `/v2`。重跑同一 summary / cold-start / SLO scorecard 到 `SERVICE_GREEN=1`、`POST_START_BLOCKED=0`、`PASS` 無 BLOCKED、all-host required observed/reachable 且 `awoooi_reboot_auto_recovery_slo_ready=1`;不可只用 route 200 或 CD `Running` 宣稱恢復。 |