diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 351c2d590..c1314205f 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -21,9 +21,10 @@ - `bash -n scripts/reboot-recovery/post-start-quick-check.sh`：通過。 - `python3.11 -m pytest scripts/reboot-recovery/tests/test_post_start_smoke_process_classifier.py scripts/reboot-recovery/tests/test_momo_source_arrival_gate.py -q`：`6 passed`。 - `git diff --check`：通過。 -- `post-reboot-readiness-summary.sh --no-color` artifact `/tmp/awoooi-post-reboot-readiness-20260629-091041-after-classifier/summary.txt`：`POST_START_PASS=43`、`POST_START_WARN=5`、`POST_START_BLOCKED=0`、`POST_START_SERVICE_WARNINGS=0`、`POST_START_BOUNDARY_WARNINGS=1`、`POST_START_EVIDENCE_WARNINGS=4`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`STOCK_FRESHNESS_STATUS=ok`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`BACKUP_CORE_GREEN=1`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=6`、`RUNTIME_ACTION_AUTHORIZED=0`、`OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`NEXT_REQUIRED_GATES=credential_escrow_evidence`。 -- `post-reboot-declaration-guard.py --summary-file /tmp/awoooi-post-reboot-readiness-20260629-091041-after-classifier/summary.txt`：`POST_REBOOT_DECLARATION_GUARD_OK status=allowed_with_boundary_blockers allowed=5 forbidden=4 next_gates=1`。 +- `post-reboot-readiness-summary.sh --no-color` artifact `/tmp/awoooi-post-reboot-readiness-20260629-091918/summary.txt`：`POST_START_PASS=43`、`POST_START_WARN=5`、`POST_START_BLOCKED=0`、`POST_START_SERVICE_WARNINGS=0`、`POST_START_BOUNDARY_WARNINGS=1`、`POST_START_EVIDENCE_WARNINGS=4`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`STOCK_FRESHNESS_STATUS=ok`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`BACKUP_CORE_GREEN=1`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=6`、`RUNTIME_ACTION_AUTHORIZED=0`、`OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`NEXT_REQUIRED_GATES=credential_escrow_evidence`。 +- `post-reboot-declaration-guard.py --summary-file /tmp/awoooi-post-reboot-readiness-20260629-091918/summary.txt`：`POST_REBOOT_DECLARATION_GUARD_OK status=allowed_with_boundary_blockers allowed=5 forbidden=4 next_gates=1 rejected_proposed=0`。 - `post-reboot-next-gate-dispatch.sh --summary-file ...`：`NEXT_GATE_COUNT=1`，唯一 gate 為 `credential_escrow_evidence`，要求 non-secret evidence id，禁止 password / token / secret value / hash / prefix / suffix / raw credential。 +- `post-reboot-owner-response-template.py --owner-packet-file /tmp/awoooi-post-reboot-owner-packets-20260629-0913.json --output /tmp/awoooi-post-reboot-owner-response-template-20260629-0913.json`：只產生 active gate `credential_escrow_evidence`，不再把已移除的 `wazuh_manager_registry_export` 帶進目前 response template；focused pytest `11 passed`，preflight 仍擋在 `blocked_waiting_owner_response_content expected_gates=1 received=0 accepted=0 runtime_gate=0`，不把模板當成批准。 - owner packet artifact `/tmp/awoooi-post-reboot-owner-packets-20260629-0913.json` 通過 contract guard：`POST_REBOOT_OWNER_PACKET_CONTRACT_GUARD_OK gates=1 request_sent=0 accepted=0 runtime_gate=0`。 - owner response preflight 正確 fail-closed：`POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_file expected_gates=1 received=0 accepted=0 runtime_gate=0`。 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index fc7896ec9..816adf30e 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,7 +1,7 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.78 -> Last updated: 2026-06-27 Asia/Taipei +> Version: v1.79 +> Last updated: 2026-06-29 Asia/Taipei > Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path. --- @@ -10,10 +10,14 @@ 本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。 -若只是重啟後要快速判斷能不能宣稱恢復，先跑機器可讀摘要：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color`。此腳本會呼叫一頁式總檢查、188 host hygiene checklist 與 Wazuh no-false-green repo gates，並把 delegated logs 和可重放的 `summary.txt` 留在 `/tmp/awoooi-post-reboot-readiness-*`。v1.75 起，同一輪驗收後續步驟必須吃同一個 `$ARTIFACT_DIR/summary.txt`，例如 `scripts/reboot-recovery/post-reboot-declaration-guard.py --summary-file "$ARTIFACT_DIR/summary.txt" --no-color` 與 `scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --summary-file "$ARTIFACT_DIR/summary.txt" --no-color`；不得在同一輪 evidence chain 反覆重跑 live probes 後混用不同時間點結論。v1.76 起，delegated cold-start 若在 K3s rollout / CD 替換瞬間出現單次 `BLOCKED AWOOOI API not reachable`，但 wrapper 自己的 public `https://awoooi.wooo.work/api/v1/health` route retry 已回 2xx，該 blocker 只列為 route/API warmup evidence warning；public API 仍失敗、其他 non-route blocker、或 retry 後未恢復時，仍維持 hard blocked。宣告 guard 會把 summary 轉成 allowed / forbidden declaration，避免把服務綠誤報成 DR complete、188 host hygiene、Wazuh registry recovered 或 runtime authorized。若 summary 顯示 `SERVICE_GREEN=1` 但 `NEXT_REQUIRED_GATES` 仍非空，再由 dispatch checklist 把尚未完成的 blocker 轉成 owner / evidence / forbidden-action checklist；需要機器可讀 intake 時，再跑 `scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --dispatch-file --output /tmp/awoooi-post-reboot-owner-packets.json` 產生 `awoooi_post_reboot_next_gate_owner_packets_v1` JSON，並立刻跑 `scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json`。dispatch / packet / guard 均固定 `DISPATCH_AUTHORIZED=0`、`REQUEST_SENT_COUNT=0`、`OWNER_RESPONSE_ACCEPTED=0`、`HOST_WRITE_AUTHORIZED=0`、`SECRET_VALUE_COLLECTION_ALLOWED=0`、`RUNTIME_GATE=0`；guard 未通過時不得送 owner request、不得寫 escrow marker、不得進維護窗口、不得宣稱 DR / Wazuh registry complete。v1.74 起，任何 owner response JSON 還必須經過 `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --owner-packet-file --response-file `：空模板、placeholder、secret payload、runtime action request、credential marker write、Wazuh active response / re-enroll / restart、Kali active scan 或缺少 Dashboard API / manager registry evidence 都必須 fail-closed；preflight 通過也只表示可進入獨立 reviewer acceptance，不是 runtime 授權。需要人工展開時，再跑 `scripts/reboot-recovery/post-start-quick-check.sh --no-color` 並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。 +若只是重啟後要快速判斷能不能宣稱恢復，先跑機器可讀摘要：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color`。此腳本會呼叫一頁式總檢查、188 host hygiene checklist 與 Wazuh no-false-green repo gates，並把 delegated logs 和可重放的 `summary.txt` 留在 `/tmp/awoooi-post-reboot-readiness-*`。v1.75 起，同一輪驗收後續步驟必須吃同一個 `$ARTIFACT_DIR/summary.txt`，例如 `scripts/reboot-recovery/post-reboot-declaration-guard.py --summary-file "$ARTIFACT_DIR/summary.txt" --no-color` 與 `scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --summary-file "$ARTIFACT_DIR/summary.txt" --no-color`；不得在同一輪 evidence chain 反覆重跑 live probes 後混用不同時間點結論。v1.76 起，delegated cold-start 若在 K3s rollout / CD 替換瞬間出現單次 `BLOCKED AWOOOI API not reachable`，但 wrapper 自己的 public `https://awoooi.wooo.work/api/v1/health` route retry 已回 2xx，該 blocker 只列為 route/API warmup evidence warning；public API 仍失敗、其他 non-route blocker、或 retry 後未恢復時，仍維持 hard blocked。宣告 guard 會把 summary 轉成 allowed / forbidden declaration，避免把服務綠誤報成 DR complete、188 host hygiene、Wazuh registry recovered 或 runtime authorized。若 summary 顯示 `SERVICE_GREEN=1` 但 `NEXT_REQUIRED_GATES` 仍非空，再由 dispatch checklist 把尚未完成的 blocker 轉成 owner / evidence / forbidden-action checklist；需要機器可讀 intake 時，再跑 `scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --summary-file "$ARTIFACT_DIR/summary.txt" --output /tmp/awoooi-post-reboot-owner-packets.json` 產生 `awoooi_post_reboot_next_gate_owner_packets_v1` JSON，立刻跑 `scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json`，並用 `scripts/reboot-recovery/post-reboot-owner-response-template.py --owner-packet-file /tmp/awoooi-post-reboot-owner-packets.json --output /tmp/awoooi-post-reboot-owner-response-template.json` 產生只含 active gates 的 placeholder response。dispatch / packet / guard / template 均固定 `DISPATCH_AUTHORIZED=0`、`REQUEST_SENT_COUNT=0`、`OWNER_RESPONSE_ACCEPTED=0`、`HOST_WRITE_AUTHORIZED=0`、`SECRET_VALUE_COLLECTION_ALLOWED=0`、`RUNTIME_GATE=0`；guard 未通過時不得送 owner request、不得寫 escrow marker、不得進維護窗口、不得宣稱 DR / Wazuh registry complete。v1.74 起，任何 owner response JSON 還必須經過 `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --owner-packet-file --response-file `：空模板、placeholder、secret payload、runtime action request、credential marker write、Wazuh active response / re-enroll / restart、Kali active scan 或缺少 active gate evidence 都必須 fail-closed；preflight 通過也只表示可進入獨立 reviewer acceptance，不是 runtime 授權。需要人工展開時，再跑 `scripts/reboot-recovery/post-start-quick-check.sh --no-color` 並以 `docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md` 作為 fallback。長 SOP 保留完整背景、例外處理與 Plan B；短版 wrapper / checklist 負責每次 T+10 分鐘內的固定判定。 v1.76 owner gate replay rule：同一輪 summary 產生後，owner packet 與 owner response preflight 必須優先使用 `--summary-file "$ARTIFACT_DIR/summary.txt"`，例如 `scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color --summary-file "$ARTIFACT_DIR/summary.txt" --output /tmp/awoooi-post-reboot-owner-packets.json` 與 `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --summary-file "$ARTIFACT_DIR/summary.txt" --response-file `。只有在刻意要重新取 live evidence 時，才允許省略 `--summary-file`；否則 preflight 不得自己重跑 summary 造成同一輪狀態漂移。 +v1.79 active owner response template rule：同一輪 owner packet 產生後，placeholder response 必須由 `scripts/reboot-recovery/post-reboot-owner-response-template.py --owner-packet-file ` 生成，讓 `responses[].gate_id` 等於 active `owner_packets[].packet_id`。目前 2026-06-29 09:13 readback 只剩 `credential_escrow_evidence`，因此 generated template 不得帶入 `wazuh_manager_registry_export`。placeholder template 必須被 preflight 擋在 `blocked_waiting_owner_response_content`、`received=0`、`accepted=0`、`runtime_gate=0`；它是 no-secret intake aid，不是 owner accepted 或 marker-write 授權。 + +2026-06-29 09:13 latest live summary：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` artifact `/tmp/awoooi-post-reboot-readiness-20260629-091918/summary.txt` 回傳 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_SERVICE_WARNINGS=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`STOCK_FRESHNESS_STATUS=ok`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`BACKUP_CORE_GREEN=1`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=6`、`RUNTIME_ACTION_AUTHORIZED=0`、`NEXT_REQUIRED_GATES=credential_escrow_evidence`。目前仍不可宣稱 `DR_COMPLETE`，因為 `ESCROW_MISSING_COUNT=5`；owner packet contract guard 期望 `gates=1`。 + 2026-06-27 11:51 最新 live revalidation：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` artifact `/tmp/awoooi-post-reboot-readiness-20260627-115046/summary.txt` 回傳 `POST_START_RESULT=BLOCKED`、`POST_START_PASS=37`、`POST_START_WARN=3`、`POST_START_BLOCKED=2`、`SERVICE_GREEN=0`、`PRODUCT_DATA_GREEN=1`、`STOCK_FRESHNESS_STATUS=ok`、`STOCK_LATEST_TRADING_DATE=2026-06-26`、`STOCK_BLOCKERS=none`、`BACKUP_CORE_GREEN=1`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`RUNTIME_ACTION_AUTHORIZED=0`。本輪已再次修復 188 `momo_pg_daily` crontab configured drift，`backup-status` 回 `core_blockers=0`、`configured_missing_188=0`；K3s / ArgoCD live readback 顯示 120 / 121 皆 `Ready`，`awoooi-prod` 為 `Synced / Healthy`，api/web/worker pods 均 Running。現在 hard blocker 是 MOMO business data freshness：`daily_sales_snapshot` 最新仍為 `2026-06-24`，`DRIVE_INTAKE_COUNT=0`，Drive archive / global latest `即時業績_當日` 均為 `2026-06-25T04:21:47Z`，最新 import job `57` 已 clean completed 且 `sync_success=true`。因此可宣稱主機、K3s、public routes、backup core 與 Stock freshness 已恢復；不可宣稱 full-stack green，直到 MOMO 來源檔補齊並由正式 import pipeline 更新 DB。DR complete 仍因 `ESCROW_MISSING_COUNT=5` 禁止宣稱，Wazuh 全主機納管仍因 manager registry accepted `0` 禁止宣稱。 2026-06-27 00:58 最新 live summary：本輪先修復兩個實際 SOP blocker。第一，`scripts/ops/recovery-scorecard-contract-check.py` 已改成 PyYAML optional，標準 Python 環境也能驗證 recovery recording-rule contract，不會因 `ModuleNotFoundError: yaml` 中斷 DR/offsite checklist。第二，188 `ollama` crontab 已備份到 `/home/ollama/momo_backups/crontab-before-momo-pg-host-owned-20260627-001925.txt`，並把 `AWOOOI momo PostgreSQL daily backup` 從 app-side `/home/ollama/momo-pro/scripts/pg_backup.sh` 收斂回 host-owned `/home/ollama/bin/momo-pg-backup.sh`；刷新 188 textfile exporter 後 `awoooi_backup_job_configured{host="188",job="momo_pg_daily"} 1`。00:58 `scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` artifact `/tmp/awoooi-post-reboot-readiness-20260627-005728/summary.txt` 回傳 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_PASS=38`、`POST_START_WARN=3`、`POST_START_BLOCKED=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`RUNTIME_ACTION_AUTHORIZED=0`。同輪 `backup-status` 回 `core_blockers=0`、`configured_missing_188=0`；Prometheus live contract 回 `awoooi_recovery_core_ready=1`、`awoooi_recovery_dr_offsite_ready=0`，表示主機 / K3s / public routes / product data / backup core 已恢復，DR 仍只因 credential escrow 缺 5 個 non-secret evidence marker blocked，Wazuh 全主機 registry accepted 仍為 0。 @@ -26,17 +30,17 @@ v1.76 owner gate replay rule：同一輪 summary 產生後，owner packet 與 ow 2026-06-26 12:13 latest live summary supersedes the 08:59 gate set：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 回傳 `POST_START_RESULT=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`POST_START_PASS=38`、`POST_START_WARN=4`、`POST_START_BLOCKED=0`、`SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`DR_ESCROW_BLOCKED=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_SERVICE_GREEN=1`、`HOST_188_HYGIENE_BLOCKED=0`、`HOST_188_RESULT=HOST_188_HYGIENE_GREEN.`、`WAZUH_ROUTE_CODE=200`、`WAZUH_TRANSPORT_COUNT=6`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`WAZUH_DASHBOARD_API_CONNECTION=pending_or_spinning`、`WAZUH_DASHBOARD_INDEX_OK=3`、`RUNTIME_ACTION_AUTHORIZED=0`、`OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`NEXT_REQUIRED_GATES=credential_escrow_evidence,wazuh_manager_registry_export`。188 host hygiene 已從 blocker 移除；目前不可宣稱完成的只剩 DR credential escrow 與 Wazuh manager registry。ACME HTTP-01 route 與 certbot timer hygiene 已修復，但不得宣稱憑證已正式 renew，需等 snap certbot timer / ACME window readback。 -2026-06-26 13:01 owner response preflight baseline：新增 `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color` 與 `docs/templates/post-reboot-next-gate-owner-response.json`。無 response file 時必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_file expected_gates=2 received=0 accepted=0 runtime_gate=0`；直接使用模板時必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_content expected_gates=2 received=0 accepted=0 runtime_gate=0`。此 gate 只驗收 `credential_escrow_evidence` 與 `wazuh_manager_registry_export` 的脫敏 owner evidence，不送 request、不寫 escrow marker、不讀 secret、不做 Wazuh / host / Kali runtime action，也不把一般批准訊息轉成 owner accepted。 +2026-06-26 13:01 owner response preflight baseline：新增 `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color` 與 `docs/templates/post-reboot-next-gate-owner-response.json`。無 response file 時必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_file expected_gates= received=0 accepted=0 runtime_gate=0`；直接使用 placeholder template 時必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_content expected_gates= received=0 accepted=0 runtime_gate=0`。2026-06-26 當時 active gate count 為 2；現行 gate count 以 v1.79 summary / owner packet 為準。此 gate 只驗收 active gates 的脫敏 owner evidence，不送 request、不寫 escrow marker、不讀 secret、不做 Wazuh / host / Kali runtime action，也不把一般批准訊息轉成 owner accepted。 2026-06-26 17:45 single-summary replay baseline：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 現在會自動寫入 `/tmp/awoooi-post-reboot-readiness-20260626-174451/summary.txt`，同一輪後續 `declaration guard`、`next-gate dispatch`、`owner packet`、`contract guard` 與 `owner response preflight` 均用此 summary 重放。17:45 summary 回傳 `SERVICE_GREEN=1`、`PRODUCT_DATA_GREEN=1`、`BACKUP_CORE_GREEN=1`、`DR_ESCROW_BLOCKED=1`、`ESCROW_MISSING_COUNT=5`、`HOST_188_HYGIENE_BLOCKED=0`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`、`NEXT_REQUIRED_GATES=credential_escrow_evidence,wazuh_manager_registry_export`。`post-start-quick-check.sh` 也已補 route warmup 分類：若 delegated cold-start 的 `BLOCKED` 全部是 public route，且 wrapper 自己的 route retry 已全部恢復，該 cold-start blocker 會降級為 evidence warning，不再把整輪服務恢復誤判成 blocked；非 route blocker 或 retry 後仍失敗仍維持 hard blocked。 2026-06-26 07:47 machine-readable readiness summary retained as historical pre-repair evidence：當時 `HOST_188_HYGIENE_BLOCKED=1`、`NEXT_REQUIRED_GATES=credential_escrow_evidence,host_188_hygiene_maintenance_window,wazuh_manager_registry_export`。此段只用來比對 188 修復前後差異；現行 gate set 必須使用 12:13 baseline。 -2026-06-26 08:12 next-gate dispatch baseline retained as historical pre-repair evidence：當時 output 固定三個 P0 checklist。12:13 起 dispatch 依 live summary 動態輸出，目前 expected `NEXT_GATE_COUNT=2`，只剩 credential escrow 與 Wazuh registry。 +2026-06-26 08:12 next-gate dispatch baseline retained as historical pre-repair evidence：當時 output 固定三個 P0 checklist。12:13 起 dispatch 依 live summary 動態輸出，當日 example 為 `NEXT_GATE_COUNT=2`、credential escrow 與 Wazuh registry；現行 gate count 必須以同一輪 summary / owner packet 為準。 2026-06-26 08:29 owner-packet JSON baseline：`scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color` 將 dispatch output 轉成 `schema_version=awoooi_post_reboot_next_gate_owner_packets_v1`，包含三個 `owner_packets`、`next_gate_count=3`、`p0_gate_count=3`、`request_sent_count=0`、`owner_response_received_count=0`、`owner_response_accepted_count=0`、`runtime_action_authorized_count=0`。此 JSON 是 AI / operator / owner review intake，不是外部 request，也不是維護窗口批准。 -2026-06-26 08:40 owner-packet contract guard baseline retained as historical pre-repair evidence：舊版鎖定三個 P0 gate。12:13 起 contract guard 依 `source.next_required_gates` 動態驗收，現行 expected success line 是 `POST_REBOOT_OWNER_PACKET_CONTRACT_GUARD_OK gates=2 request_sent=0 accepted=0 runtime_gate=0`；若 188 hygiene future regression，才會回到 `gates=3`。 +2026-06-26 08:40 owner-packet contract guard baseline retained as historical pre-repair evidence：舊版鎖定三個 P0 gate。12:13 起 contract guard 依 `source.next_required_gates` 動態驗收；2026-06-26 artifact 當時是 `POST_REBOOT_OWNER_PACKET_CONTRACT_GUARD_OK gates=2 request_sent=0 accepted=0 runtime_gate=0`，現行 expected line 以同一輪 summary / owner packet 為準。若 188 hygiene future regression，才會重新增加對應 gate。 2026-06-26 08:47 Wazuh registry detail baseline：`scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` 已把 Wazuh repo-side coverage / runtime gate 的細節納入固定 key/value：`WAZUH_COVERAGE_SCOPE=6`、`WAZUH_DIRECT_ACTIVE=2`、`WAZUH_NO_TRANSPORT=1`、`WAZUH_SSH_BLOCKED=3`、`WAZUH_ROUTE_CODE=200`、`WAZUH_TRANSPORT_COUNT=6`、`WAZUH_DASHBOARD_API_CONNECTION=pending_or_spinning`、`WAZUH_DASHBOARD_INDEX_OK=3`、`WAZUH_MANAGER_REGISTRY_ACCEPTED=0`、`WAZUH_RUNTIME_GATE=0`。`scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color` 的 `wazuh_manager_registry_export` gate 會把這些狀態放入 `CURRENT_EVIDENCE`。判讀鐵律：route `200`、transport `6`、Dashboard index pattern `3` 都不是 manager registry accepted；全主機納管與 Dashboard API 修復仍需 owner evidence / registry export / acceptance record。 diff --git a/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md b/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md index d45c62e32..05f6bdbc3 100644 --- a/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md +++ b/docs/runbooks/REBOOT-POST-START-QUICK-CHECK.md @@ -94,7 +94,7 @@ scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color --summary-file "$SUMMARY_FILE" | tee /tmp/awoooi-post-reboot-dispatch.txt ``` -這支腳本只把目前 live summary 內的 `NEXT_REQUIRED_GATES` 轉成 owner / required evidence / forbidden action / done criteria。2026-06-26 12:13 起通常只剩 `credential_escrow_evidence` 與 `wazuh_manager_registry_export`；若未來 188 又紅，才會重新出現 `host_188_hygiene_maintenance_window`。它不送 request、不讀 secret、不寫 marker、不 restart / reload / repair / import / delete / patch，也不授權 host / Wazuh / Nginx / K8s / DB runtime action。若它輸出 `SERVICE_GREEN=0`，先回到服務恢復，不進入 boundary dispatch。 +這支腳本只把目前 live summary 內的 `NEXT_REQUIRED_GATES` 轉成 owner / required evidence / forbidden action / done criteria。gate set 必須以同一份 `summary.txt` 為準；2026-06-29 09:13 readback 只剩 `credential_escrow_evidence`。它不送 request、不讀 secret、不寫 marker、不 restart / reload / repair / import / delete / patch，也不授權 host / Wazuh / Nginx / K8s / DB runtime action。若它輸出 `SERVICE_GREEN=0`，先回到服務恢復，不進入 boundary dispatch。 若要交給 AI / 工單 / owner review 使用，產生機器可讀 owner packet： @@ -111,17 +111,18 @@ scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color --summ scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json ``` -guard 必須輸出 `POST_REBOOT_OWNER_PACKET_CONTRACT_GUARD_OK gates= request_sent=0 accepted=0 runtime_gate=0`。目前預期 `gates=2`；若 188 hygiene 回到 blocked，才會是 `gates=3`。若 gate 數量、P0 gate id、`0 / false` 欄位、禁用 secret payload、Wazuh 禁用 active response / host write，或 no-false-green 規則任何一項漂移，視為 `BLOCKED`，不得送 owner request、不得寫 escrow marker、不得進維護窗口、不得宣稱 DR / Wazuh 完成。 +guard 必須輸出 `POST_REBOOT_OWNER_PACKET_CONTRACT_GUARD_OK gates= request_sent=0 accepted=0 runtime_gate=0`。目前 2026-06-29 readback 預期 `gates=1`；若 Wazuh registry 或 188 hygiene 回到 active next gate，gate count 才會依 summary 增加。若 gate 數量、P0 gate id、`0 / false` 欄位、禁用 secret payload、Wazuh 禁用 active response / host write，或 no-false-green 規則任何一項漂移，視為 `BLOCKED`，不得送 owner request、不得寫 escrow marker、不得進維護窗口、不得宣稱 DR / Wazuh 完成。 收到 owner response 檔案前，或收到任何聲稱已補證據的 JSON 前，必須跑 owner response preflight： ```bash scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --summary-file "$ARTIFACT_DIR/summary.txt" scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --owner-packet-file /tmp/awoooi-post-reboot-owner-packets.json -scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --owner-packet-file /tmp/awoooi-post-reboot-owner-packets.json --response-file docs/templates/post-reboot-next-gate-owner-response.json +scripts/reboot-recovery/post-reboot-owner-response-template.py --owner-packet-file /tmp/awoooi-post-reboot-owner-packets.json --output /tmp/awoooi-post-reboot-owner-response-template.json +scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color --owner-packet-file /tmp/awoooi-post-reboot-owner-packets.json --response-file /tmp/awoooi-post-reboot-owner-response-template.json ``` -第一個命令必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_file expected_gates=2 received=0 accepted=0 runtime_gate=0`。第二個命令必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_content expected_gates=2 received=0 accepted=0 runtime_gate=0`，證明空模板不能被算成已收件或已接受。合格 response 只能包含脫敏 evidence refs、owner role / team / decision / reviewer / followup owner、五個 escrow item 的 non-secret evidence ref，以及 Wazuh manager registry / Dashboard API readback；不得包含密碼、token、secret value、hash、prefix/suffix、raw Wazuh payload、agent 原名、內網 IP、`client.keys`、active response、host write、agent re-enroll、Wazuh restart、Kali active scan 或 credential marker write。preflight 通過也只代表可進入獨立 reviewer acceptance，不代表 `DR_COMPLETE`、`WAZUH_REGISTRY_RECOVERED` 或任何 runtime action 授權。 +前兩個 preflight 命令必須輸出 `POST_REBOOT_OWNER_RESPONSE_PREFLIGHT_BLOCKED status=blocked_waiting_owner_response_file expected_gates= received=0 accepted=0 runtime_gate=0`。generator 只產生 active gate 的 placeholder JSON；目前 2026-06-29 readback 只應包含 `credential_escrow_evidence`，不得額外帶入 `wazuh_manager_registry_export`。把 placeholder template 送回 preflight 時，必須輸出 `status=blocked_waiting_owner_response_content`、`received=0`、`accepted=0`、`runtime_gate=0`，證明空模板不能被算成已收件或已接受。合格 response 只能包含 active gate 要求的脫敏 evidence refs、owner role / team / decision / reviewer / followup owner、五個 escrow item 的 non-secret evidence ref；若 Wazuh gate 未來重新 active，才納入 Wazuh manager registry / Dashboard API readback。不得包含密碼、token、secret value、hash、prefix/suffix、raw Wazuh payload、agent 原名、內網 IP、`client.keys`、active response、host write、agent re-enroll、Wazuh restart、Kali active scan 或 credential marker write。preflight 通過也只代表可進入獨立 reviewer acceptance，不代表 `DR_COMPLETE`、`WAZUH_REGISTRY_RECOVERED` 或任何 runtime action 授權。 需要展開細節時，再使用 repo-side wrapper： diff --git a/docs/templates/post-reboot-next-gate-owner-response.json b/docs/templates/post-reboot-next-gate-owner-response.json index 23f0f5efe..d4b83b2ed 100644 --- a/docs/templates/post-reboot-next-gate-owner-response.json +++ b/docs/templates/post-reboot-next-gate-owner-response.json @@ -1,5 +1,9 @@ { "schema_version": "awoooi_post_reboot_next_gate_owner_response_v1", + "template_status": "fallback_example_only_use_post_reboot_owner_response_template_py_for_active_gates", + "active_gates": [ + "credential_escrow_evidence" + ], "responses": [ { "gate_id": "credential_escrow_evidence", @@ -13,10 +17,13 @@ ], "followup_owner": "followup_owner_here", "runtime_action_requested": false, + "runtime_action_authorized": false, "host_write_requested": false, + "host_write_authorized": false, "secret_value_included": false, "secret_value_collection_allowed": false, "credential_marker_write_requested": false, + "credential_marker_write_authorized": false, "escrow_items": [ { "item_id": "restic_repository_password", @@ -59,40 +66,6 @@ "contains_secret_value": false } ] - }, - { - "gate_id": "wazuh_manager_registry_export", - "owner_role": "owner_role_here", - "owner_team": "owner_team_here", - "decision": "pending", - "decision_reason": "decision_reason_here", - "affected_scope": "Wazuh manager registry redacted export", - "redacted_evidence_refs": [ - "redacted_evidence_ref_here" - ], - "followup_owner": "followup_owner_here", - "runtime_action_requested": false, - "host_write_requested": false, - "secret_value_included": false, - "secret_value_collection_allowed": false, - "wazuh_active_response_requested": false, - "agent_reenroll_requested": false, - "wazuh_restart_requested": false, - "kali_active_scan_requested": false, - "registry_export_ref": "registry_export_ref_here", - "registry_time_window": "pending", - "expected_host_aliases": [ - "core-110", - "gateway-188", - "k3s-control-120", - "k3s-control-121", - "security-observer-112", - "dev-workstation-111" - ], - "manager_registry_count": 0, - "dashboard_api_connection_status": "pending", - "dashboard_api_version_status": "pending", - "reviewer": "reviewer_here" } ] } diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index b50640065..cc6adea5f 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -15,19 +15,21 @@ | P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; latest 2026-06-26 07:19 readback shows 120 and 121 reachable, K3s active, `mon` and `mon1` both `Ready control-plane`, AWOOOI API/Web replicas split across both nodes, ArgoCD `awoooi-prod Synced / Healthy` at revision `1fd5e2a8b0f18d24eed16aa2a44286bcbf230603`, and `km-vectorize` official 03:00 台北時間 run succeeded with `lastSuccess=2026-06-25T19:00:14Z`. | | P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 98% | 2026-06-27 00:56 backup readback shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `integrity_stale=0`, `offsite_fresh=1`, `rclone_gdrive_fresh=1`, `configured_missing_188=0`, `escrow_missing=5`, last aggregate `2026-06-26 02:31:02`。188 MOMO backup crontab drift 已修復並保留 rollback crontab。DR remains blocked on real non-secret credential escrow evidence IDs; do not write placeholder markers or paste secret values. | | P2 service / data truth | DONE | 100% | Public routes 與 service health 為綠燈，MOMO health `V10.719`，current-month parity 為 `15383|15383|2026-06-01|2026-06-24|2026-06-01|2026-06-24`。StockPlatform `/api/v1/system/freshness` 為 `ok`，latest trading date `2026-06-26`，blockers `none`；先前 Stock EOD blocker 已由官方來源與正式 cron 自然收斂。 | -| P3 docs / automation contracts | DONE_WITH_BACKUP_CORE_RECOVERY_V178 | 100% | Workplan, SOP v1.78, post-reboot declaration guard, machine-readable post-reboot readiness summary with Wazuh registry detail fields and auto-persisted `summary.txt`, post-reboot next-gate dispatch checklist, owner-packet JSON generator, dynamic owner-packet contract guard, post-reboot owner response preflight, owner response placeholder template, one-page post-start quick check v1.18, route retry gate, delegated cold-start public-route / AWOOOI API warmup classifier, backup-status core-blocker readback, PyYAML-optional recovery-scorecard contract check, 188 MOMO backup crontab host-owned rollback evidence, deploy warmup classification, expanded public route list, StockPlatform freshness gate, StockPlatform cron-source recovery evidence, StockPlatform natural schedule green evidence, 110 orphan Chrome recurrence cleanup evidence, 188 fail-closed startup data recovery gate, 188 host hygiene read-only checklist, 188 PostgreSQL runtime-ready source-of-truth, 188 ACME route/timer hygiene, baseline `stockplatform_system_freshness_ok`, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat suppression, MOMO scheduler / current-month detector fix, exporter restore helpers, 110 Docker disk pressure cleanup boundary, notification-noise readback, MOMO import-boundary / Drive-auth fail-closed deploys, product version/readback matrix, and stricter product-data / route retry gates are updated. Declaration guard now machine-checks allowed / forbidden recovery statements from the same `summary.txt`: service/data/backup/188 host hygiene green may be declared when live summary says so, while `DR_COMPLETE`、`WAZUH_REGISTRY_RECOVERED` and `RUNTIME_ACTION_AUTHORIZED` remain forbidden until evidence gates close. | +| P3 docs / automation contracts | DONE_WITH_BACKUP_CORE_RECOVERY_V179 | 100% | Workplan, SOP v1.79, post-reboot declaration guard, machine-readable post-reboot readiness summary with Wazuh registry detail fields and auto-persisted `summary.txt`, post-reboot next-gate dispatch checklist, owner-packet JSON generator, dynamic owner-packet contract guard, post-reboot owner response preflight, active-gate owner response template generator, one-page post-start quick check v1.18, route retry gate, delegated cold-start public-route / AWOOOI API warmup classifier, backup-status core-blocker readback, PyYAML-optional recovery-scorecard contract check, 188 MOMO backup crontab host-owned rollback evidence, deploy warmup classification, expanded public route list, StockPlatform freshness gate, StockPlatform cron-source recovery evidence, StockPlatform natural schedule green evidence, 110 orphan Chrome recurrence cleanup evidence, 188 fail-closed startup data recovery gate, 188 host hygiene read-only checklist, 188 PostgreSQL runtime-ready source-of-truth, 188 ACME route/timer hygiene, baseline `stockplatform_system_freshness_ok`, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, explicit Plan B degraded-operation path, machine-readable `plan_b` baseline, readiness-audit Plan B guard, B0-B5 service levels, T+0/T+120 fallback timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, `fwupd-refresh.timer` rollback note, K3s filesystem event blocker, AWOOOI backup no-direct-offsite-sync contract, 110/188 Ansible source-of-truth, Gitea self-hosted readiness validation workflow, post-CD no-regression readbacks, stale-vs-active K8s failed Job classification, 110 runaway browser / CI load AIOps exporter + alert + gated remediation PlayBook, Telegram / AI event packet mapping, healthy heartbeat suppression, MOMO scheduler / current-month detector fix, exporter restore helpers, 110 Docker disk pressure cleanup boundary, notification-noise readback, MOMO import-boundary / Drive-auth fail-closed deploys, product version/readback matrix, and stricter product-data / route retry gates are updated. Declaration guard now machine-checks allowed / forbidden recovery statements from the same `summary.txt`: service/data/backup/188 host hygiene green may be declared when live summary says so, while `DR_COMPLETE`、`WAZUH_REGISTRY_RECOVERED` and `RUNTIME_ACTION_AUTHORIZED` remain forbidden until evidence gates close. | 2026-06-26 12:13 machine-readable summary baseline supersedes the 07:47 / 08:59 gate set: `scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` stores delegated logs under `/tmp/awoooi-post-reboot-readiness-20260626-121303` and returns `SERVICE_GREEN=1`, `PRODUCT_DATA_GREEN=1`, `BACKUP_CORE_GREEN=1`, `DR_ESCROW_BLOCKED=1`, `ESCROW_MISSING_COUNT=5`, `HOST_188_SERVICE_GREEN=1`, `HOST_188_HYGIENE_BLOCKED=0`, `HOST_188_CHECK_RC=0`, `HOST_188_RESULT=HOST_188_HYGIENE_GREEN.`, `WAZUH_ROUTE_CODE=200`, `WAZUH_TRANSPORT_COUNT=6`, `WAZUH_COVERAGE_SCOPE=6`, `WAZUH_DIRECT_ACTIVE=2`, `WAZUH_NO_TRANSPORT=1`, `WAZUH_SSH_BLOCKED=3`, `WAZUH_DASHBOARD_API_CONNECTION=pending_or_spinning`, `WAZUH_DASHBOARD_INDEX_OK=3`, `WAZUH_MANAGER_REGISTRY_ACCEPTED=0`, `WAZUH_RUNTIME_GATE=0`, `RUNTIME_ACTION_AUTHORIZED=0`, `OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, and `NEXT_REQUIRED_GATES=credential_escrow_evidence,wazuh_manager_registry_export`. This is now the preferred first operator/AI-agent entrypoint after reboot because it separates service health from DR and security registry evidence; 188 host hygiene is no longer a next gate unless the live checklist regresses. 2026-06-26 07:47 machine-readable summary baseline is retained as historical evidence only. It showed `HOST_188_HYGIENE_BLOCKED=1` and three next gates before the 188 startup / ACME / certbot hygiene repair. Do not use the 07:47 gate set as the current status. -2026-06-26 12:13 next-gate dispatch baseline: `scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color` now emits only the gates present in the current summary. Current expected gates are `credential_escrow_evidence` and `wazuh_manager_registry_export`, with `NEXT_GATE_COUNT=2`, `REQUEST_SENT_COUNT=0`, `DISPATCH_AUTHORIZED=0`, `HOST_WRITE_AUTHORIZED=0`, `SECRET_VALUE_COLLECTION_ALLOWED=0`, `RUNTIME_ACTION_AUTHORIZED=0`. If 188 hygiene regresses, `host_188_hygiene_maintenance_window` will reappear automatically. +2026-06-26 12:13 next-gate dispatch baseline: `scripts/reboot-recovery/post-reboot-next-gate-dispatch.sh --no-color` now emits only the gates present in the current summary. 2026-06-26 example gates were `credential_escrow_evidence` and `wazuh_manager_registry_export`, with `NEXT_GATE_COUNT=2`, `REQUEST_SENT_COUNT=0`, `DISPATCH_AUTHORIZED=0`, `HOST_WRITE_AUTHORIZED=0`, `SECRET_VALUE_COLLECTION_ALLOWED=0`, `RUNTIME_ACTION_AUTHORIZED=0`; current gate count must follow the same-run summary / owner packet, and 2026-06-29 expects only `credential_escrow_evidence`. If 188 hygiene regresses, `host_188_hygiene_maintenance_window` will reappear automatically. 2026-06-26 12:13 owner-packet JSON baseline: `scripts/reboot-recovery/post-reboot-next-gate-owner-packets.py --no-color` emits `schema_version=awoooi_post_reboot_next_gate_owner_packets_v1` with dynamic `next_gate_count=2`, `p0_gate_count=2`, `request_sent_count=0`, `owner_response_received_count=0`, `owner_response_accepted_count=0`, `runtime_action_authorized_count=0`. This packet is for AI / operator / owner review intake only; it does not send request, write credential marker, read secret, or authorize runtime action. -2026-06-26 12:13 owner-packet contract guard baseline: `scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json` validates the generated JSON before any owner review intake. It requires the packet gates to equal the live `source.next_required_gates`, preserves `request_sent=0`、`owner_response_received=0`、`owner_response_accepted=0`、`runtime_action_authorized=0`、`host_write_authorized=0`、`secret_value_collection_allowed=0`、`runtime_gate=0`, and rejects missing forbidden payload/action controls for active gates. Current expected success line: `POST_REBOOT_OWNER_PACKET_CONTRACT_GUARD_OK gates=2 request_sent=0 accepted=0 runtime_gate=0`. +2026-06-26 12:13 owner-packet contract guard baseline: `scripts/reboot-recovery/post-reboot-owner-packet-contract-guard.py --packet-file /tmp/awoooi-post-reboot-owner-packets.json` validates the generated JSON before any owner review intake. It requires the packet gates to equal the live `source.next_required_gates`, preserves `request_sent=0`、`owner_response_received=0`、`owner_response_accepted=0`、`runtime_action_authorized=0`、`host_write_authorized=0`、`secret_value_collection_allowed=0`、`runtime_gate=0`, and rejects missing forbidden payload/action controls for active gates. 2026-06-26 artifact 當時是 two-gate example；現行 gate count 必須以同一輪 summary / owner packet 為準，2026-06-29 artifact 預期 `POST_REBOOT_OWNER_PACKET_CONTRACT_GUARD_OK gates=1 request_sent=0 accepted=0 runtime_gate=0`。 -2026-06-26 13:01 owner response preflight baseline: `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color` validates future owner responses against the dynamic owner-packet gate set without sending requests, writing markers, reading secrets, or changing runtime. Missing response file must remain `blocked_waiting_owner_response_file`; the placeholder template `docs/templates/post-reboot-next-gate-owner-response.json` must remain `blocked_waiting_owner_response_content` with `received=0`, `accepted=0`, and `runtime_gate=0`. The only acceptable payload class is redacted owner evidence for credential escrow and Wazuh manager registry export; secret values, hash / prefix / suffix, raw Wazuh payload, agent real names, internal IPs, `client.keys`, credential marker write, host write, Wazuh active response / re-enroll / restart, and Kali active scan are rejected. +2026-06-29 09:13 active owner response template baseline: latest summary artifact `/tmp/awoooi-post-reboot-readiness-20260629-091918/summary.txt` returns `NEXT_REQUIRED_GATES=credential_escrow_evidence`, so `post-reboot-next-gate-owner-packets.py --summary-file ...` and the contract guard expect `gates=1`. `scripts/reboot-recovery/post-reboot-owner-response-template.py --owner-packet-file ` now generates placeholder responses from the active owner packet instead of relying on a static two-gate JSON. The generated placeholder must remain `blocked_waiting_owner_response_content` with `expected_gates=1`, `received=0`, `accepted=0`, and `runtime_gate=0`; it is no-secret intake scaffolding only, not owner acceptance, not request sent, and not credential marker write authorization. + +2026-06-26 13:01 owner response preflight baseline: `scripts/reboot-recovery/post-reboot-owner-response-preflight.py --no-color` validates future owner responses against the dynamic owner-packet gate set without sending requests, writing markers, reading secrets, or changing runtime. Missing response file must remain `blocked_waiting_owner_response_file`; placeholder templates must remain `blocked_waiting_owner_response_content` with `received=0`, `accepted=0`, and `runtime_gate=0`. The only acceptable payload class is redacted owner evidence for currently active gates; secret values, hash / prefix / suffix, raw Wazuh payload, agent real names, internal IPs, `client.keys`, credential marker write, host write, Wazuh active response / re-enroll / restart, and Kali active scan are rejected. 2026-06-26 17:45 single-summary replay baseline: `scripts/reboot-recovery/post-reboot-readiness-summary.sh --no-color` now writes the exact emitted key/value summary to `$ARTIFACT_DIR/summary.txt`; latest artifact `/tmp/awoooi-post-reboot-readiness-20260626-174451/summary.txt` returns `SERVICE_GREEN=1`, `PRODUCT_DATA_GREEN=1`, `BACKUP_CORE_GREEN=1`, `DR_ESCROW_BLOCKED=1`, `ESCROW_MISSING_COUNT=5`, `HOST_188_HYGIENE_BLOCKED=0`, `WAZUH_MANAGER_REGISTRY_ACCEPTED=0`, `RUNTIME_ACTION_AUTHORIZED=0`, `OVERALL_DECLARATION=FULL_STACK_GREEN_DR_ESCROW_BLOCKED`, and `NEXT_REQUIRED_GATES=credential_escrow_evidence,wazuh_manager_registry_export`. The same summary file drives declaration guard, next-gate dispatch, owner packet generation, contract guard, and owner response preflight. `post-start-quick-check.sh` now holds delegated cold-start blockers until wrapper route retry completes; route-only cold-start blockers that recover under wrapper retry are evidence warnings, while non-route blockers or unrecovered routes remain hard blockers. diff --git a/scripts/reboot-recovery/post-reboot-owner-response-template.py b/scripts/reboot-recovery/post-reboot-owner-response-template.py new file mode 100755 index 000000000..d1ff76f43 --- /dev/null +++ b/scripts/reboot-recovery/post-reboot-owner-response-template.py @@ -0,0 +1,262 @@ +#!/usr/bin/env python3 +"""Generate a fail-closed owner response template for active post-reboot gates. + +Read-only by design. This script only turns an owner-packet artifact into a +placeholder response JSON. It never sends requests, reads secrets, writes +credential markers, or modifies host/runtime state. +""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +from datetime import datetime +from pathlib import Path +from typing import Any + + +ROOT = Path(__file__).resolve().parents[2] +OWNER_PACKET_GENERATOR = ( + ROOT / "scripts" / "reboot-recovery" / "post-reboot-next-gate-owner-packets.py" +) + +EXPECTED_OWNER_PACKET_SCHEMA = "awoooi_post_reboot_next_gate_owner_packets_v1" +RESPONSE_SCHEMA = "awoooi_post_reboot_next_gate_owner_response_v1" + +ESCROW_ITEM_IDS = [ + "restic_repository_password", + "offsite_provider_credentials", + "break_glass_admin_credentials", + "dns_registrar_recovery", + "oauth_ai_provider_recovery", +] + +EXPECTED_HOST_ALIASES = [ + "core-110", + "gateway-188", + "k3s-control-120", + "k3s-control-121", + "security-observer-112", + "dev-workstation-111", +] + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Create a placeholder owner response JSON for active post-reboot gates.", + ) + parser.add_argument( + "--owner-packet-file", + type=Path, + help="Use an existing post-reboot owner packet JSON.", + ) + parser.add_argument( + "--dispatch-file", + type=Path, + help="Generate owner packets from an existing next-gate dispatch file.", + ) + parser.add_argument( + "--summary-file", + type=Path, + help="Generate owner packets from an existing readiness summary file.", + ) + parser.add_argument( + "--output", + type=Path, + help="Write JSON to this path instead of stdout.", + ) + parser.add_argument( + "--no-color", + action="store_true", + help="Pass --no-color when generating owner packets.", + ) + return parser.parse_args() + + +def load_json(path: Path, label: str) -> dict[str, Any]: + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except FileNotFoundError as exc: + raise SystemExit(f"{label}_not_found={path}") from exc + except json.JSONDecodeError as exc: + raise SystemExit(f"{label}_json_invalid={exc}") from exc + if not isinstance(payload, dict): + raise SystemExit(f"{label}_json_not_object") + return payload + + +def generate_owner_packet(args: argparse.Namespace) -> dict[str, Any]: + cmd = [str(OWNER_PACKET_GENERATOR)] + if args.dispatch_file: + cmd.extend(["--dispatch-file", str(args.dispatch_file)]) + if args.summary_file: + cmd.extend(["--summary-file", str(args.summary_file)]) + if args.no_color: + cmd.append("--no-color") + + completed = subprocess.run( + cmd, + cwd=ROOT, + check=False, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + ) + if completed.returncode != 0: + raise SystemExit( + "owner_packet_generation_failed " + f"rc={completed.returncode}\n{completed.stdout}" + ) + try: + packet = json.loads(completed.stdout) + except json.JSONDecodeError as exc: + raise SystemExit(f"owner_packet_json_invalid={exc}") from exc + if not isinstance(packet, dict): + raise SystemExit("owner_packet_json_not_object") + return packet + + +def load_owner_packet(args: argparse.Namespace) -> dict[str, Any]: + if args.owner_packet_file: + return load_json(args.owner_packet_file, label="owner_packet_file") + return generate_owner_packet(args) + + +def as_list(value: Any) -> list[Any]: + if value is None: + return [] + if isinstance(value, list): + return value + return [value] + + +def ensure_owner_packet(packet: dict[str, Any]) -> list[dict[str, Any]]: + if packet.get("schema_version") != EXPECTED_OWNER_PACKET_SCHEMA: + raise SystemExit(f"owner_packet_schema={packet.get('schema_version')!r}") + owner_packets = [item for item in as_list(packet.get("owner_packets")) if isinstance(item, dict)] + source = packet.get("source", {}) + if isinstance(source, dict): + expected = {str(item) for item in as_list(source.get("next_required_gates"))} + actual = {str(item.get("packet_id")) for item in owner_packets if item.get("packet_id")} + if expected and actual != expected: + raise SystemExit(f"owner_packet_gate_mismatch expected={sorted(expected)} actual={sorted(actual)}") + return owner_packets + + +def common_response(packet: dict[str, Any]) -> dict[str, Any]: + return { + "gate_id": str(packet.get("packet_id", "unknown")), + "owner_role": "owner_role_here", + "owner_team": "owner_team_here", + "decision": "pending", + "decision_reason": "decision_reason_here", + "affected_scope": str(packet.get("title", "post-reboot owner evidence")), + "redacted_evidence_refs": [ + "redacted_evidence_ref_here", + ], + "followup_owner": "followup_owner_here", + "runtime_action_requested": False, + "runtime_action_authorized": False, + "host_write_requested": False, + "host_write_authorized": False, + "secret_value_included": False, + "secret_value_collection_allowed": False, + } + + +def credential_escrow_response(packet: dict[str, Any]) -> dict[str, Any]: + item_ids = [ + str(item) + for item in as_list(packet.get("required_items")) + if str(item) in ESCROW_ITEM_IDS + ] or ESCROW_ITEM_IDS + response = common_response(packet) + response["credential_marker_write_requested"] = False + response["credential_marker_write_authorized"] = False + response["escrow_items"] = [ + { + "item_id": item_id, + "non_secret_evidence_ref": "non_secret_evidence_ref_here", + "recovery_owner": "owner_role_here", + "reviewer": "reviewer_here", + "last_reviewed_at": "pending", + "contains_secret_value": False, + } + for item_id in item_ids + ] + return response + + +def wazuh_registry_response(packet: dict[str, Any]) -> dict[str, Any]: + response = common_response(packet) + response.update( + { + "wazuh_active_response_requested": False, + "wazuh_active_response_authorized": False, + "agent_reenroll_requested": False, + "wazuh_restart_requested": False, + "kali_active_scan_requested": False, + "registry_export_ref": "registry_export_ref_here", + "registry_time_window": "pending", + "expected_host_aliases": EXPECTED_HOST_ALIASES, + "manager_registry_count": 0, + "dashboard_api_connection_status": "pending", + "dashboard_api_version_status": "pending", + "reviewer": "reviewer_here", + } + ) + return response + + +def generic_unsupported_response(packet: dict[str, Any]) -> dict[str, Any]: + response = common_response(packet) + response["gate_specific_evidence_required"] = [ + "update_post_reboot_owner_response_preflight_for_this_gate", + ] + return response + + +def response_for(packet: dict[str, Any]) -> dict[str, Any]: + gate_id = str(packet.get("packet_id", "")) + if gate_id == "credential_escrow_evidence": + return credential_escrow_response(packet) + if gate_id == "wazuh_manager_registry_export": + return wazuh_registry_response(packet) + return generic_unsupported_response(packet) + + +def build_template(packet: dict[str, Any]) -> dict[str, Any]: + owner_packets = ensure_owner_packet(packet) + gate_ids = [ + str(item.get("packet_id")) + for item in owner_packets + if item.get("packet_id") + ] + return { + "schema_version": RESPONSE_SCHEMA, + "generated_at": datetime.now().astimezone().isoformat(timespec="seconds"), + "template_status": "placeholder_fail_closed_until_redacted_owner_evidence_is_supplied", + "source_owner_packet_schema": packet.get("schema_version", "unknown"), + "active_gate_count": len(gate_ids), + "active_gates": gate_ids, + "responses": [response_for(item) for item in owner_packets], + } + + +def main() -> int: + args = parse_args() + packet = load_owner_packet(args) + template = build_template(packet) + payload = json.dumps(template, ensure_ascii=False, indent=2, sort_keys=True) + if args.output: + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(payload + "\n", encoding="utf-8") + else: + print(payload) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/reboot-recovery/tests/test_post_reboot_owner_response_template.py b/scripts/reboot-recovery/tests/test_post_reboot_owner_response_template.py new file mode 100644 index 000000000..236ff2612 --- /dev/null +++ b/scripts/reboot-recovery/tests/test_post_reboot_owner_response_template.py @@ -0,0 +1,117 @@ +from __future__ import annotations + +import json +import subprocess +import sys +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[3] +TEMPLATE_SCRIPT = ROOT / "scripts" / "reboot-recovery" / "post-reboot-owner-response-template.py" +PREFLIGHT_SCRIPT = ROOT / "scripts" / "reboot-recovery" / "post-reboot-owner-response-preflight.py" + + +ESCROW_ITEMS = [ + "restic_repository_password", + "offsite_provider_credentials", + "break_glass_admin_credentials", + "dns_registrar_recovery", + "oauth_ai_provider_recovery", +] + + +def write_packet(tmp_path: Path, gate_ids: list[str]) -> Path: + owner_packets = [] + for gate_id in gate_ids: + packet = { + "packet_id": gate_id, + "title": f"{gate_id} owner evidence", + "priority": "P0", + "required_items": ESCROW_ITEMS if gate_id == "credential_escrow_evidence" else [], + } + owner_packets.append(packet) + + packet_path = tmp_path / "owner-packets.json" + packet_path.write_text( + json.dumps( + { + "schema_version": "awoooi_post_reboot_next_gate_owner_packets_v1", + "source": {"next_required_gates": gate_ids}, + "owner_packets": owner_packets, + }, + indent=2, + ) + + "\n", + encoding="utf-8", + ) + return packet_path + + +def run_template(packet_path: Path, output_path: Path | None = None) -> dict: + cmd = [ + sys.executable, + str(TEMPLATE_SCRIPT), + "--owner-packet-file", + str(packet_path), + ] + if output_path: + cmd.extend(["--output", str(output_path)]) + + result = subprocess.run(cmd, text=True, capture_output=True, check=True) + if output_path: + return json.loads(output_path.read_text(encoding="utf-8")) + return json.loads(result.stdout) + + +def test_generates_only_active_credential_gate(tmp_path: Path) -> None: + packet_path = write_packet(tmp_path, ["credential_escrow_evidence"]) + template = run_template(packet_path) + + assert template["schema_version"] == "awoooi_post_reboot_next_gate_owner_response_v1" + assert template["active_gates"] == ["credential_escrow_evidence"] + assert [item["gate_id"] for item in template["responses"]] == ["credential_escrow_evidence"] + assert "wazuh_manager_registry_export" not in json.dumps(template) + + response = template["responses"][0] + assert response["runtime_action_requested"] is False + assert response["runtime_action_authorized"] is False + assert response["host_write_requested"] is False + assert response["host_write_authorized"] is False + assert response["secret_value_included"] is False + assert response["secret_value_collection_allowed"] is False + assert response["credential_marker_write_requested"] is False + assert response["credential_marker_write_authorized"] is False + assert [item["item_id"] for item in response["escrow_items"]] == ESCROW_ITEMS + assert all(item["contains_secret_value"] is False for item in response["escrow_items"]) + + +def test_placeholder_template_fails_closed_against_current_preflight(tmp_path: Path) -> None: + packet_path = write_packet(tmp_path, ["credential_escrow_evidence"]) + response_path = tmp_path / "owner-response-template.json" + run_template(packet_path, output_path=response_path) + + result = subprocess.run( + [ + sys.executable, + str(PREFLIGHT_SCRIPT), + "--owner-packet-file", + str(packet_path), + "--response-file", + str(response_path), + "--json", + ], + text=True, + capture_output=True, + check=True, + ) + preflight = json.loads(result.stdout) + + assert preflight["status"] == "blocked_waiting_owner_response_content" + assert preflight["expected_gate_count"] == 1 + assert preflight["expected_gates"] == ["credential_escrow_evidence"] + assert preflight["owner_response_received_count"] == 0 + assert preflight["owner_response_accepted_count"] == 0 + assert preflight["runtime_gate_count"] == 0 + assert preflight["runtime_action_authorized"] is False + assert not any("unknown_gate_ids" in blocker for blocker in preflight["blockers"]) + assert any("credential_escrow_evidence.owner_role_missing" in blocker for blocker in preflight["blockers"])