diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 0b3a0ae27..05b26e3a5 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -54296,3 +54296,26 @@ production browser smoke: **下一步**: - 跑 full focused verifier / runner pressure guard / `git diff --check` 後 commit / push 到 Gitea main;等待 CD deploy marker 後讀回 production priority / SLO,並接續取得 111 VMX source external verified backup artifact 或更強 identity evidence,再做 scoped relink / restore dry-run。 + +## 2026-07-03 — P0-006 Windows99 VMX locator via-host closure + +**完成內容**: +- `scripts/reboot-recovery/locate-windows99-vmx-source.sh` 新增 `--via-host` / `WINDOWS99_VIA_HOST` 支援,讓官方 locator 可從目前工作站經 110 no-secret SSH path 執行,而不是卡在本機直連 99 public-key auth missing。 +- live 執行 `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect --via-host wooo@192.168.0.110 --users wooo --timeout 8 --max-auth-users 1` 成功:`locator_collection_status=collected_windows99_vmx_source_locator_stdout`、`expected_vmx_path_present_count=0`、`candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`。 +- locator 輸出維持 public-safe / no-secret 邊界:`secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`、`raw_path_output=false`、`path_fingerprint_only=true`。結果比前一個 identity metadata scanner 更保守,只輸出 fingerprint,不輸出 raw path。 +- SOP 已更新 v1.116 / v5.8:下一步不是猜 generic Ubuntu candidate,而是取得既有 VMX source 或 verified backup path,然後做 scoped relink / restore dry-run + post-verifier。 + +**本地驗證結果**: +- `bash -n scripts/reboot-recovery/locate-windows99-vmx-source.sh`:通過。 +- `python3.11 -m pytest scripts/reboot-recovery/tests/test_windows99_vmx_source_locator.py -q -p no:cacheprovider`:`5 passed`。 +- `python3.11 -m pytest scripts/reboot-recovery/tests/test_windows99_vmx_source_locate_check.py scripts/reboot-recovery/tests/test_windows99_vmx_source_locator.py -q -p no:cacheprovider`:`7 passed`。 +- `DATABASE_URL=postgresql+asyncpg://test:test@localhost/test PYTHONPATH=apps/api python3.11 -m pytest apps/api/tests/test_windows99_vmx_source_locator_readback_api.py apps/api/tests/test_awoooi_priority_work_order_readback_api.py -q -p no:cacheprovider`:`21 passed`。 +- `python3.11 ops/runner/guard-gitea-runner-pressure.py --root .`、`git diff --check`:通過。 + +**仍維持**: +- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth。 +- 沒有使用 GitHub / gh / GitHub API / GitHub Actions。 +- 沒有重啟主機,沒有 Docker / Nginx / K3s / DB / firewall restart,沒有 workflow_dispatch,沒有 DROP / TRUNCATE / restore / prune,沒有啟動或關閉 VM。 + +**下一步**: +- commit / push 到 Gitea main;deploy 後讀回 production locator endpoint 與 priority readback。接續 P0 是提供 / 找到 111 verified VMX source 或 backup path,再產生 scoped dry-run,不直接 relink 或 power on。 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 92177809e..a0380e333 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,6 +1,6 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.115 +> Version: v1.116 > Last updated: 2026-07-03 Asia/Taipei > Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號,但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal;此納入不代表授權任何破壞性 runtime apply。 @@ -62,6 +62,8 @@ v1.114 Windows99 VMX package public redaction boundary rule:public `/api/v1/ag v1.115 Windows99 VMX source locate check-mode rule:2026-07-03 12:34 deploy 後 production readback 已更新為 `status=blocked_reboot_auto_recovery_slo_not_ready`、`active_blocker_count=10`、`readiness_percent=60`、`primary_blocker=reboot_event_required_host_unreachable`、`can_claim_all_services_recovered_within_target=false`;priority readback 同步顯示 `windows99_vmx_source_repair_action_stage=source_repair_check_mode_ready`、`windows99_vmx_source_repair_next_executable_step=locate_or_relink_missing_vmx_source_check_mode`、`windows99_vmx_source_locator_status=check_mode_locator_ready_windows99_vmx_source_required`,locator endpoint 顯示 `missing_vmx_aliases=["111"]` 與 safe command `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect`。定位 111 VMX source 時固定使用 no-secret locator / collector;本輪 live check-mode 結果:expected 111 VMX path `exists=0`、`candidate_vmx_count=6`、`alias_hint_candidate_count=0`、`identity_target_hint_count=0`、`identity_unassigned_ubuntu_candidate_count=1`、`locate_status=single_identity_unassigned_ubuntu_candidate_requires_confirmation`。該 generic Ubuntu candidate 不能直接 relink;先前 MAC / neighbor evidence 未證明它是 111,且 99 local named backup roots 與 110 bounded backup roots 未找到 111 VMX artifact。下一步只能是取得外部 verified backup artifact 或更強身份證據後建立 scoped relink / restore dry-run,再跑 post-verifier;不得用猜測候選直接 relink、power on 或宣稱 SLA 完成。 +v1.116 Windows99 VMX locator via-host execution rule:若目前工作站直連 99 缺 public-key auth,但 110 到 99 的 no-secret SSH path 已驗證,官方 locator 必須支援 `--via-host`,固定可執行命令為 `bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect --via-host wooo@192.168.0.110 --users wooo --max-auth-users 1`。2026-07-03 live readback:`locator_collection_status=collected_windows99_vmx_source_locator_stdout`、`expected_vmx_path_present_count=0`、`candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`、`safe_next_step=provide_existing_vmx_source_or_backup_path_then_rerun_locator_check_mode_no_write`,且 `secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`、`raw_path_output=false`、`path_fingerprint_only=true`。此結果把下一步收斂為「提供既有 VMX source 或 verified backup path」;仍不得 relink generic VM、啟動 VM、restore 到 production 或讀密碼。 + 2026-07-02 110 control-path / Harbor recovery receipt rule:若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure,但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`,則該 Gitea Harbor repair 失敗只能列為 historical queue metadata,不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier,並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示:public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`;active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure,以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker,non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker;non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。 2026-07-02 110 controlled CD lane fail-closed enforcer staging rule:110 runner 壓力事故後,legacy / generic runner 仍必須 fail-closed;但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail,且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時,才可保留 drain config / binary / unit,並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service;若 registration 缺失,readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走,優先修 source enforcer / unit guardrail,不要手工反覆補同一組 artifact。 diff --git a/docs/runbooks/REBOOT-RECOVERY-SOP.md b/docs/runbooks/REBOOT-RECOVERY-SOP.md index 37595e2c7..206d9eac0 100644 --- a/docs/runbooks/REBOOT-RECOVERY-SOP.md +++ b/docs/runbooks/REBOOT-RECOVERY-SOP.md @@ -46,7 +46,7 @@ - 188 可達,但 `systemd_state=degraded` 且 `awoooi-startup.service failed`。 - `reboot_event_required_host_unreachable` 仍為 primary blocker,因此 10 分鐘 SLO 尚未證明。 -固定下一步:`locate_or_relink_missing_vmx_source_check_mode`。production `/api/v1/agents/windows99-vmx-source-locator-readback` 已回 `status=check_mode_locator_ready_windows99_vmx_source_required`、`missing_vmx_aliases=["111"]`、`safe_next_step=bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect`、`secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_allowed=false`。本輪 live check-mode 結果為 expected 111 VMX path `exists=0`、`candidate_vmx_count=6`、`alias_hint_candidate_count=0`、`identity_target_hint_count=0`、`identity_unassigned_ubuntu_candidate_count=1`、`locate_status=single_identity_unassigned_ubuntu_candidate_requires_confirmation`。唯一 generic Ubuntu candidate 還不能安全 relink,因為 display name / alias 沒有 111 identity hint,先前 MAC / neighbor evidence 也未證明它是 111,且 99 local named backup roots 與 110 bounded backup roots 尚未找到 111 VMX artifact。下一步只能取得外部 verified backup artifact 或更強 identity evidence,再產生 scoped relink / restore dry-run 與 post-verifier。若 public API response 顯示 `public_lan_topology_redacted=true` 或 `check_mode_probe_commands_executable=false`,其中 `host:...` VMX path 只能當 public display evidence,不得直接貼到 Windows console 執行;要執行 probe 時必須使用授權內部 console / no-secret package 保留的原始 Windows path。禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。 +固定下一步:`locate_or_relink_missing_vmx_source_check_mode`。production `/api/v1/agents/windows99-vmx-source-locator-readback` 已回 `status=check_mode_locator_ready_windows99_vmx_source_required`、`missing_vmx_aliases=["111"]`、`safe_next_step=bash scripts/reboot-recovery/locate-windows99-vmx-source.sh --collect`、`secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_allowed=false`。目前工作站必須加 `--via-host wooo@192.168.0.110 --users wooo --max-auth-users 1` 走已驗證的 110 no-secret path;live 結果為 `locator_collection_status=collected_windows99_vmx_source_locator_stdout`、`expected_vmx_path_present_count=0`、`candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`、`raw_path_output=false`、`path_fingerprint_only=true`。下一步只能取得既有 VMX source 或 external verified backup path,再產生 scoped relink / restore dry-run 與 post-verifier。若 public API response 顯示 `public_lan_topology_redacted=true` 或 `check_mode_probe_commands_executable=false`,其中 `host:...` VMX path 只能當 public display evidence,不得直接貼到 Windows console 執行;要執行 probe 時必須使用授權內部 console / no-secret package 保留的原始 Windows path。禁止讀 Windows 密碼、啟動 / 關閉 VM、host reboot、service restart、Docker / Nginx / K3s / DB / firewall restart、restore、prune、delete。 ### 五主機全貌 diff --git a/scripts/reboot-recovery/locate-windows99-vmx-source.sh b/scripts/reboot-recovery/locate-windows99-vmx-source.sh index a4b3bf6f9..49726768d 100755 --- a/scripts/reboot-recovery/locate-windows99-vmx-source.sh +++ b/scripts/reboot-recovery/locate-windows99-vmx-source.sh @@ -8,6 +8,8 @@ CONNECT_TIMEOUT="${WINDOWS99_CONNECT_TIMEOUT:-3}" SSH_TIMEOUT="${WINDOWS99_SSH_TIMEOUT:-3}" REMOTE_LOCATOR_TIMEOUT="${WINDOWS99_REMOTE_LOCATOR_TIMEOUT:-60}" SSH_PORT="${WINDOWS99_SSH_PORT:-22}" +VIA_HOST="${WINDOWS99_VIA_HOST:-}" +VIA_HOST_PORT="${WINDOWS99_VIA_PORT:-22}" MAX_AUTH_USERS="${WINDOWS99_MAX_AUTH_USERS:-}" MAX_AUTH_USERS_EXPLICIT=0 if [[ -n "${WINDOWS99_MAX_AUTH_USERS:-}" ]]; then @@ -30,7 +32,7 @@ is_positive_int() { } usage() { - printf '%s\n' "usage: $0 [--check|--collect] [--host HOST] [--users 'u1 u2'] [--timeout SECONDS] [--max-auth-users N]" + printf '%s\n' "usage: $0 [--check|--collect] [--host HOST] [--users 'u1 u2'] [--via-host USER@HOST] [--timeout SECONDS] [--max-auth-users N]" } while [[ $# -gt 0 ]]; do @@ -51,6 +53,14 @@ while [[ $# -gt 0 ]]; do SSH_USERS=(${1:-}) SSH_USERS_EXPLICIT=1 ;; + --via-host) + shift + VIA_HOST="${1:-}" + ;; + --via-port) + shift + VIA_HOST_PORT="${1:-22}" + ;; --timeout) shift CONNECT_TIMEOUT="${1:-5}" @@ -175,15 +185,48 @@ SSH_OPTS=( -p "${SSH_PORT}" ) +VIA_SSH_OPTS=( + -o BatchMode=yes + -o PreferredAuthentications=publickey + -o PubkeyAuthentication=yes + -o PasswordAuthentication=no + -o KbdInteractiveAuthentication=no + -o NumberOfPasswordPrompts=0 + -o ConnectTimeout="${SSH_TIMEOUT}" + -o ConnectionAttempts=1 + -o GSSAPIAuthentication=no + -o LogLevel=ERROR + -o StrictHostKeyChecking=no + -o UserKnownHostsFile="${KNOWN_HOSTS_FILE}" + -p "${VIA_HOST_PORT}" +) + +quote_words() { + printf '%q ' "$@" +} + run_ssh_timed() { local user="$1" local timeout_seconds="$2" local timeout_wrapper_seconds="$((timeout_seconds + 1))" + local inner_command="" shift 2 if ! is_positive_int "${timeout_seconds}"; then timeout_seconds="${SSH_TIMEOUT}" timeout_wrapper_seconds="$((SSH_TIMEOUT + 1))" fi + if [[ -n "${VIA_HOST}" ]]; then + local inner_ssh=(ssh "${SSH_OPTS[@]}" -o ConnectTimeout="${timeout_seconds}" "${user}@${TARGET_HOST}" "$@") + inner_command="$(quote_words "${inner_ssh[@]}")" + if [[ "${SSH_TIMEOUT_WRAPPER}" == "timeout" ]]; then + timeout "${timeout_wrapper_seconds}s" ssh "${VIA_SSH_OPTS[@]}" "${VIA_HOST}" "bash -lc $(printf '%q' "${inner_command}")" + elif [[ "${SSH_TIMEOUT_WRAPPER}" == "gtimeout" ]]; then + gtimeout "${timeout_wrapper_seconds}s" ssh "${VIA_SSH_OPTS[@]}" "${VIA_HOST}" "bash -lc $(printf '%q' "${inner_command}")" + else + ssh "${VIA_SSH_OPTS[@]}" "${VIA_HOST}" "bash -lc $(printf '%q' "${inner_command}")" + fi + return $? + fi if [[ "${SSH_TIMEOUT_WRAPPER}" == "timeout" ]]; then timeout "${timeout_wrapper_seconds}s" ssh "${SSH_OPTS[@]}" -o ConnectTimeout="${timeout_seconds}" "${user}@${TARGET_HOST}" "$@" elif [[ "${SSH_TIMEOUT_WRAPPER}" == "gtimeout" ]]; then @@ -277,6 +320,7 @@ fi printf '%s\n' "schema_version=windows99_vmx_source_locator_collector_v1" printf '%s\n' "dry_run=${DRY_RUN}" printf '%s\n' "target_host=${TARGET_HOST}" +printf '%s\n' "via_host=${VIA_HOST:-none}" printf '%s\n' "target_host_alias=99" printf '%s\n' "target_vm_alias=111" printf '%s\n' "connect_timeout_seconds=${CONNECT_TIMEOUT}" diff --git a/scripts/reboot-recovery/tests/test_windows99_vmx_source_locator.py b/scripts/reboot-recovery/tests/test_windows99_vmx_source_locator.py index ed6db376f..8ca4029f3 100644 --- a/scripts/reboot-recovery/tests/test_windows99_vmx_source_locator.py +++ b/scripts/reboot-recovery/tests/test_windows99_vmx_source_locator.py @@ -50,6 +50,8 @@ def test_locator_contract_forbids_secret_runtime_actions_and_vmx_content_reads() assert "PasswordAuthentication=no" in bash_text assert "KbdInteractiveAuthentication=no" in bash_text assert "NumberOfPasswordPrompts=0" in bash_text + assert "--via-host" in bash_text + assert "via_host=" in bash_text assert "in_memory_stdin_scriptblock" in bash_text assert "[Console]::In.ReadToEnd()" in bash_text assert "Get-ChildItem" in ps_text @@ -163,7 +165,7 @@ def test_collect_mode_accepts_valid_remote_locator_stdout(tmp_path: Path) -> Non """ #!/usr/bin/env bash joined="$*" - if [[ "$joined" == *"echo AWOOOI_WINDOWS99_SSH_READY"* ]]; then + if [[ "$joined" == *"AWOOOI_WINDOWS99_SSH_READY"* ]]; then printf '%s\n' 'AWOOOI_WINDOWS99_SSH_READY' exit 0 fi @@ -197,3 +199,65 @@ def test_collect_mode_accepts_valid_remote_locator_stdout(tmp_path: Path) -> Non assert "remote_locator_output_begin" in result.stdout assert "source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found" in result.stdout assert "remote_locator_output_end" in result.stdout + + +def test_collect_mode_supports_no_secret_via_host_path(tmp_path: Path) -> None: + fake_bin = tmp_path / "bin" + fake_bin.mkdir() + _write_executable( + fake_bin / "nc", + """ + #!/usr/bin/env bash + [[ "${!#}" == "22" ]] + """, + ) + _write_executable( + fake_bin / "ssh", + """ + #!/usr/bin/env bash + joined="$*" + if [[ "$joined" == *"AWOOOI_WINDOWS99_SSH_READY"* ]]; then + printf '%s\n' 'AWOOOI_WINDOWS99_SSH_READY' + exit 0 + fi + if [[ "$joined" == *"powershell"* ]]; then + cat >/dev/null + printf '%s\n' 'EXPECTED_VMX alias=111 index=1 present=0 fingerprint=abc' + printf '%s\n' 'AWOOOI_WINDOWS99_VMX_SOURCE_LOCATOR=1' + printf '%s\n' 'schema_version=windows99_vmx_source_locator_v1' + printf '%s\n' 'MODE=Check' + printf '%s\n' 'target_host_alias=99' + printf '%s\n' 'target_vm_alias=111' + printf '%s\n' 'expected_vmx_path_count=1' + printf '%s\n' 'expected_vmx_path_present_count=0' + printf '%s\n' 'candidate_source_count=0' + printf '%s\n' 'source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found' + printf '%s\n' 'file_content_read=false' + printf '%s\n' 'remote_write_performed=false' + exit 0 + fi + exit 255 + """, + ) + + result = _run_locator( + fake_bin, + "--collect", + "--via-host", + "wooo@192.168.0.110", + "--users", + "wooo", + "--max-auth-users", + "1", + ) + + assert result.returncode == 0 + values = _key_values(result.stdout) + assert values["via_host"] == "wooo@192.168.0.110" + assert values["remote_locator_attempted"] == "1" + assert values["locator_collection_status"] == ( + "collected_windows99_vmx_source_locator_stdout" + ) + assert values["secret_value_read"] == "false" + assert values["remote_write_performed"] == "false" + assert values["vm_power_change_performed"] == "false"