diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index c8c35a745..6b1d800b9 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,25 @@ +## 2026-07-09 — P0-006 AI log triage controlled auto-recovery package + +**完成內容**: +- 新增 `scripts/reboot-recovery/ai-log-triage-auto-recovery.sh`,把 110 Docker / containerd / Harbor / Nginx、120 / 121 K3s events、188 edge Nginx / Docker / systemd、99 VMware no-secret readback 與 public route status 收斂成 `findings.tsv`、`actions.tsv`、`summary.txt`。 +- `reboot-auto-recovery-slo-exporter.sh` 會在 host probe / readiness summary / scorecard 前先跑 AI log triage;`awoooi-reboot-auto-recovery-slo.service` 設定 `AI_LOG_TRIAGE_AUTO_RECOVERY_MODE=apply`,但 allowlist 僅限 start inactive docker/nginx、Harbor watchdog log-first repair、registry ready 後刪 ImagePullBackOff / ErrImagePull pods。 +- 新增 full-host reboot orchestrator 與 systemd unit / installer;installer 預設 `--dry-run`,必須明確 `--apply --enable-now` 才會啟用 timer,unit 使用 `/opt/awoooi/reboot-recovery` portable path,避免 Mac local path 進 production。 +- 修正 orchestrator:`ai-log-triage-auto-recovery.sh` 以 exit `2` 表示 P0 findings 時不再被誤判成 `AI_LOG_TRIAGE_FAILED`,會正確 merge findings 到 orchestrator summary。 +- 新增 Windows99 source scripts:VMware autostart 與 Windows Update surprise reboot prevention;兩者保留 `WhatIfOnly` / source contract,不在本輪執行 Windows write。 + +**已跑驗證**: +- `PYTHONPATH=apps/api python3.11 -m pytest scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py -q -p no:cacheprovider`:`10 passed`。 +- `bash -n scripts/reboot-recovery/ai-log-triage-auto-recovery.sh scripts/reboot-recovery/full-host-reboot-orchestrator.sh scripts/reboot-recovery/install-full-host-reboot-orchestrator.sh scripts/reboot-recovery/awoooi-telegram-notify.sh scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh`:通過。 +- `python3.11 -m py_compile scripts/reboot-recovery/reboot-auto-recovery-slo-scorecard.py scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py`、`git diff --check`:通過。 +- Live `--check` artifact `/tmp/awoooi-full-host-reboot-orchestrator-codex/20260709T082305Z`:`RECOVERY_BLOCKED_P0`、`FINDING_COUNT=3`、`FINDING_P0_COUNT=2`,P0 為 `STOCK_UPSTREAM_NOT_READY` 與 `EDGE_502_UPSTREAM_REFUSED`;public route readback 為 AWOOOI health `200`、AWOOOI web `307`、Stock freshness `200`、Gitea `200`、registry `/v2/` `401`、Harbor `200`。 + +**仍維持**: +- Production P0-006 尚未完成:`/api/v1/agents/reboot-auto-recovery-slo-scorecard` 仍為 `blocked_reboot_auto_recovery_slo_not_ready`,最新讀回 `active_blocker_count=16`、primary `reboot_event_required_host_unreachable`,next safe action 仍是 `restore_windows99_missing_vmx_source_for_aliases_then_rerun_no_secret_collector_and_scorecard_no_vm_power_change`。 +- 本輪未執行 `--apply`、未啟用 timer、未重啟 host / Docker / Nginx / K3s / DB / firewall、未 VM power change、未讀 secret / token / `.env` / raw sessions / SQLite / auth、未使用 GitHub / gh、未 DROP / TRUNCATE / restore / prune / delete / force push。 + +**下一步**: +- 跑完整 focused verifier / runner pressure guard;commit / normal push Gitea main;CD 後讀回 production SLO / priority work order,確認 AI log triage source controls 與 orchestrator evidence 上線。P0-006 下一個 runtime 收斂仍要先取得 verified 111 VMX source / backup path 或授權 console source path,再進 scoped dry-run,不能猜路徑或直接 power on VM。 + ## 2026-07-03 — 21:21 全站 UI/UX 專業化重構 — 面板與頁面 Tailwind 化 **完成內容**: diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 270a7c2a7..24b9a1716 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -10,7 +10,7 @@ 本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天,必須先重跑 live check,再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。 -v1.131 AI log triage controlled auto-recovery rule:每次全主機重啟後,10 分鐘 SLO verifier 不得只跑 read-only summary。`awoooi-reboot-auto-recovery-slo.service` 必須以 `AI_LOG_TRIAGE_AUTO_RECOVERY_MODE=apply` 先執行 `scripts/reboot-recovery/ai-log-triage-auto-recovery.sh`,bounded 收集 99 / 110 / 120 / 121 / 188 logs、K3s events、VMware no-secret readback 與 public route status,自動輸出 findings / actions / summary,再接 host probe、post-reboot summary、scorecard 與 textfile metrics。2026-07-09 16:17 live check artifact `/tmp/awoooi-ai-log-triage-codex-20260709-161759` 的目前 truth 是 `FINDING_COUNT=4`、`FINDING_P0_COUNT=3`:`TOO_MANY_OPEN_FILES host=110_or_188`、`STOCK_UPSTREAM_NOT_READY host=110_188`、`EDGE_502_UPSTREAM_REFUSED host=188_edge`,另有 `HOST188_POSTGRES_SYSTEMD_FAILED` P2 hygiene;public route readback 為 AWOOOI health `200`、AWOOOI web `307`、Stock freshness `200`、Gitea `200`、registry `/v2/` `401`、Harbor `200`。這些不得再被當成噪音或人工 SOP;必須進 AI findings、Prometheus/Telegram blocker 與下一步 action matrix。允許的 `--apply` 只限低風險 allowlist:start inactive docker/nginx、Harbor watchdog log-first repair、registry ready 後刪 failed-pull pods 讓 K3s 重建。仍禁止 reboot、node drain、firewall cutover、DB destructive action、Docker prune、secret/session/.env/raw-volume reads。 +v1.131 AI log triage controlled auto-recovery rule:每次全主機重啟後,10 分鐘 SLO verifier 不得只跑 read-only summary。`awoooi-reboot-auto-recovery-slo.service` 必須以 `AI_LOG_TRIAGE_AUTO_RECOVERY_MODE=apply` 先執行 `scripts/reboot-recovery/ai-log-triage-auto-recovery.sh`,bounded 收集 99 / 110 / 120 / 121 / 188 logs、K3s events、VMware no-secret readback 與 public route status,自動輸出 findings / actions / summary,再接 host probe、post-reboot summary、scorecard 與 textfile metrics。2026-07-09 16:23 orchestrator check artifact `/tmp/awoooi-full-host-reboot-orchestrator-codex/20260709T082305Z` 的目前 truth 是 `RECOVERY_BLOCKED_P0`、`FINDING_COUNT=3`、`FINDING_P0_COUNT=2`:`STOCK_UPSTREAM_NOT_READY host=110_188`、`EDGE_502_UPSTREAM_REFUSED host=188_edge`,另有 `HOST188_POSTGRES_SYSTEMD_FAILED` P2 hygiene;public route readback 為 AWOOOI health `200`、AWOOOI web `307`、Stock freshness `200`、Gitea `200`、registry `/v2/` `401`、Harbor `200`。這些不得再被當成噪音或人工 SOP;必須進 AI findings、Prometheus/Telegram blocker 與下一步 action matrix。允許的 `--apply` 只限低風險 allowlist:start inactive docker/nginx、Harbor watchdog log-first repair、registry ready 後刪 failed-pull pods 讓 K3s 重建。仍禁止 reboot、node drain、firewall cutover、DB destructive action、Docker prune、secret/session/.env/raw-volume reads。 v1.130 Windows99 VMX backup/source search package rule:2026-07-03 18:19 最新 via-110 no-secret locator 已把目前 truth 從早先 `candidate_source_count=6` 修正為 `candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`,且 `raw_path_output=false`、`path_fingerprint_only=true`、`secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`。因此 P0-4 下一步不得再引用 generic Ubuntu candidate;必須讀 `/api/v1/agents/windows99-vmx-source-backup-search-package` 與 priority readback 的 `windows99_vmx_source_backup_search_*` 欄位。當前 backup/source search receipt 顯示 120 / 121 / local 均未找到 verified 111 source,99 alias 無 Linux backup host mapping,110 / 188 / 112 SSH BatchMode 不可用,`candidate_source_count=0`、`blocked_host_count=4`、`apply_allowed_by_this_package=false`。固定下一步是離線備份 inventory 或授權 console source path,再取得 verified source path 後建立 scoped relink dry-run;仍不得 power on VM、restore、讀 VMX 內容、讀 secret、重啟 Windows / VMware service / host / Docker / Nginx / K3s / DB / firewall。 diff --git a/docs/runbooks/REBOOT-RECOVERY-SOP.md b/docs/runbooks/REBOOT-RECOVERY-SOP.md index da28bcf34..608d3a52b 100644 --- a/docs/runbooks/REBOOT-RECOVERY-SOP.md +++ b/docs/runbooks/REBOOT-RECOVERY-SOP.md @@ -25,7 +25,7 @@ 主機重啟後不得再靠人工逐台看 log 才判斷異常。固定第一步改為執行 `scripts/reboot-recovery/ai-log-triage-auto-recovery.sh --apply`;若只允許讀取,則用 `--check`,但 scorecard / 告警仍必須吃同一份 artifact。此 runner 會 bounded 收集 110 Docker / containerd / Harbor / Nginx、120 / 121 K3s events、188 edge Nginx / Docker / systemd、99 VMware no-secret readback 與 public route HTTP status,然後輸出 `findings.tsv`、`actions.tsv`、`summary.txt`。 -本輪 2026-07-09 16:17 live evidence `/tmp/awoooi-ai-log-triage-codex-20260709-161759` 顯示 `FINDING_COUNT=4`、`FINDING_P0_COUNT=3`,P0 依序為 `TOO_MANY_OPEN_FILES host=110_or_188`、`STOCK_UPSTREAM_NOT_READY host=110_188`、`EDGE_502_UPSTREAM_REFUSED host=188_edge`,另有 `HOST188_POSTGRES_SYSTEMD_FAILED` P2 hygiene。public route readback 為 AWOOOI health `200`、AWOOOI web `307`、Stock freshness `200`、Gitea `200`、registry `/v2/` `401`、Harbor `200`;因此不得沿用稍早 Harbor 5000 refused / K3s ImagePullBackOff 舊 artifact 當作目前 truth。若後續 registry 再度不 ready,K3s pod delete 只能被自動化標為 `skipped_registry_not_ready`,不得宣稱 502 已修復。 +本輪 2026-07-09 16:23 orchestrator check evidence `/tmp/awoooi-full-host-reboot-orchestrator-codex/20260709T082305Z` 顯示 `RECOVERY_BLOCKED_P0`、`FINDING_COUNT=3`、`FINDING_P0_COUNT=2`,P0 依序為 `STOCK_UPSTREAM_NOT_READY host=110_188`、`EDGE_502_UPSTREAM_REFUSED host=188_edge`,另有 `HOST188_POSTGRES_SYSTEMD_FAILED` P2 hygiene。public route readback 為 AWOOOI health `200`、AWOOOI web `307`、Stock freshness `200`、Gitea `200`、registry `/v2/` `401`、Harbor `200`;因此不得沿用稍早 Harbor 5000 refused / K3s ImagePullBackOff 舊 artifact 當作目前 truth。若後續 registry 再度不 ready,K3s pod delete 只能被自動化標為 `skipped_registry_not_ready`,不得宣稱 502 已修復。 `awoooi-reboot-auto-recovery-slo.service` 必須設定 `AI_LOG_TRIAGE_AUTO_RECOVERY_MODE=apply`,並由 `reboot-auto-recovery-slo-exporter.sh` 在 host probe / readiness summary 前先跑 AI log triage。允許的 controlled apply 僅限:110 Docker inactive 時 start、Harbor watchdog `--repair-once` / harbor-log-first recovery、registry ready 後只刪 `ImagePullBackOff` / `ErrImagePull` pods 讓 Kubernetes 重建、188 nginx/docker inactive 時 start。仍禁止 reboot、node drain、firewall cutover、DB restore/drop/truncate、Docker prune、secret/session/.env/raw-volume reads。 diff --git a/scripts/reboot-recovery/full-host-reboot-orchestrator.sh b/scripts/reboot-recovery/full-host-reboot-orchestrator.sh index 6dccbb139..c643c222e 100755 --- a/scripts/reboot-recovery/full-host-reboot-orchestrator.sh +++ b/scripts/reboot-recovery/full-host-reboot-orchestrator.sh @@ -202,15 +202,20 @@ probe_public_routes() { run_ai_log_triage() { local triage_dir="${ARTIFACT_DIR}/ai-log-triage" + local rc=0 if [[ ! -x "${SCRIPT_DIR}/ai-log-triage-auto-recovery.sh" ]]; then finding "P0" "recovery" "AI_LOG_TRIAGE_SCRIPT_MISSING" "${SCRIPT_DIR}/ai-log-triage-auto-recovery.sh" return 1 fi action "$MODE" "ai-log-triage" "RUN_AI_LOG_TRIAGE" "artifact_dir=${triage_dir}" - "${SCRIPT_DIR}/ai-log-triage-auto-recovery.sh" "--${MODE}" --artifact-dir "$triage_dir" || { + "${SCRIPT_DIR}/ai-log-triage-auto-recovery.sh" "--${MODE}" --artifact-dir "$triage_dir" || rc=$? + if [[ "$rc" -ne 0 && "$rc" -ne 2 ]]; then finding "P0" "ai-log-triage" "AI_LOG_TRIAGE_FAILED" "see ${triage_dir}" return 1 - } + fi + if [[ "$rc" -eq 2 ]]; then + event "INFO" "ai-log-triage" "AI_LOG_TRIAGE_REPORTED_FINDINGS rc=2" + fi if [[ -s "${triage_dir}/findings.tsv" ]]; then awk -v now="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -F '\t' \ '{print now "\t" $2 "\tai-log-triage\t" $1 "\thost=" $3 " evidence=" $4 " next_action=" $5}' \ diff --git a/scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py b/scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py index 0f1337dc4..3f3a301d8 100644 --- a/scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py +++ b/scripts/reboot-recovery/tests/test_reboot_p0_operational_contract.py @@ -194,6 +194,9 @@ def test_reboot_p0_contract_has_ai_log_triage_controlled_apply() -> None: assert "run_scorecard" in full_host_orchestrator assert "run_stock_upstream_gate_hint" in full_host_orchestrator assert "TELEGRAM_NOTIFY_SCRIPT" in full_host_orchestrator + assert 'rc=$?' in full_host_orchestrator + assert '"$rc" -ne 2' in full_host_orchestrator + assert "AI_LOG_TRIAGE_REPORTED_FINDINGS" in full_host_orchestrator assert 'host=" $3 " evidence=" $4 " next_action=" $5' in full_host_orchestrator assert "/Users/ogt/" not in full_host_service assert "WorkingDirectory=/opt/awoooi/reboot-recovery" in full_host_service