diff --git a/AGENT99-AI-WORKSTATION-SOP.md b/AGENT99-AI-WORKSTATION-SOP.md index 6ad3e7317..ac4b69da7 100644 --- a/AGENT99-AI-WORKSTATION-SOP.md +++ b/AGENT99-AI-WORKSTATION-SOP.md @@ -709,3 +709,38 @@ Agent99 now accepts operator text instructions and turns them into allowlisted q - Evidence: `C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-224134.json`. - Public smoke: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502. - Telegram result: `operator_command_result`, `severity=info`, `sent=true`, `relayOk=true`. + +## 2026-07-10 M9 Source-Trusted Transport And Verified Remediation Receipt + +Agent99 now uses a source-trusted deployment path, a dedicated bounded SSH transport, and verifier-authoritative remediation outcomes. + +- Source authority and deployment: + - Gitea commit SHA is the source revision; a local scratch checkout or a Windows file timestamp is not deployment truth. + - `agent99-deploy.ps1` validates PowerShell parse, Agent99 contracts, synthetic checks, staging content, backup, atomic promotion, task registration, and rollback. + - `C:\Wooo\Agent99\runtime-manifest.json.sourceRevision` must match the deployed Gitea SHA. A successful source test without this runtime readback is not a completed deployment. +- Dedicated transport: + - Private identity: `C:\Wooo\Agent99\keys\agent99_ed25519`; it remains on 99, is ACL-restricted, and is never copied into evidence, Telegram, logs, or the repository. + - Public-key authorization is limited with `from="192.168.0.99",restrict` on 110 user `wooo` and 112 user `kali`. + - Route map: 110 and 112 direct; 120, 121, and 188 prefer 110 as the jump host. + - Live public-key fingerprint receipt: `SHA256:4ccZZSe0buoWyivi2mfTMXE9LZfCmFBPr/evCzZ0nzA`. +- Automatic recovery: + - Agent99 persists boot identity in `C:\Wooo\Agent99\state\boot-state.json`. + - A changed boot observation queues one Recover request only; duplicate requests are suppressed by single-flight state and a 10-minute cooldown. + - `recovery_slo_result` records the recovery window. It may claim success only after a real reboot drill proves all required hosts, services, routes, and freshness checks inside 10 minutes. +- Verifier-authoritative remediation: + - When an action configures `postVerify`, command success alone is insufficient. The verifier result is the effective action result. + - A failed verifier creates only the 10-minute failure backoff; it cannot create a success cooldown. + - `policyVersion` migration updates stale live policies and prevents an old command or marker from surviving a new source deployment. +- Bounded SSH process lifecycle: + - Every SSH call has a timeout; timeout cleanup terminates the SSH process tree and verifies that the client exited. + - Before controlled `Perf` or `LoadShed`, the stale-process guard only considers Agent99 allowlisted targets using `BatchMode=yes` and explicitly excludes `-N`, `-L`, `-R`, and `-D` tunnels. + - A process is eligible after 15 minutes with a missing parent, or after the 60-minute hard limit. Evidence records selected process metadata but never the command line. +- Live receipts: + - Key provisioning: `C:\Wooo\Agent99\evidence\agent99-ssh-key-provision-final.json`. + - False-success marker correction: `C:\Wooo\Agent99\evidence\agent99-load-shed-marker-correction.json`. + - Controlled stale-client cleanup: `C:\Wooo\Agent99\evidence\agent99-stale-ssh-cleanup-v2.json`; 7 orphan clients stopped. + - Performance readback: `C:\Wooo\Agent99\evidence\agent99-Perf-20260710-224454.json`; all five hosts OK, 188 disk reduced from 90% to 88%. + - The deployment receipt and runtime manifest must contain the final deployed Gitea SHA; paths or source-test success alone are not accepted as runtime proof. +- Remaining acceptance truth: + - This receipt closes the transport stall, orphan SSH, 188 disk pressure, and false-green remediation defects. + - It does not close the whole-host 10-minute cold-start objective until the next controlled reboot produces a fresh scorecard with all required gates green. diff --git a/agent99-contract-check.ps1 b/agent99-contract-check.ps1 index 0b197c145..f09f39759 100644 --- a/agent99-contract-check.ps1 +++ b/agent99-contract-check.ps1 @@ -66,6 +66,7 @@ $bootstrap = [IO.File]::ReadAllText((Join-Path $SourceRoot "agent99-bootstrap.ps Add-Check "transport:dedicated_identity" ($control.Contains("sshIdentityFile") -and $control.Contains("IdentitiesOnly=yes")) "dedicated SSH identity is enforced" Add-Check "transport:jump_first" ($control.Contains("preferJumpHost") -and $control.Contains("ssh_jump_fallback")) "preferred jump with bounded fallback is present" +Add-Check "transport:stale_process_guard" ($control.Contains("Invoke-AgentStaleSshProcessGuard") -and $control.Contains("taskkill.exe /PID") -and $control.Contains("ssh_stale_process_guard")) "timed-out and stale SSH clients are bounded" Add-Check "recovery:auto_trigger" ($control.Contains("Get-AgentBootState") -and $control.Contains("Start-AgentRecoveryFromObservation") -and $control.Contains("recovery_already_queued")) "boot and host-down recovery use a single-flight queue" Add-Check "recovery:slo" ($control.Contains("recovery_slo_result") -and $control.Contains("withinSlo")) "10-minute recovery evidence is present" Add-Check "outcome:verified" ($control.Contains("Get-AgentOutcomeContract") -and $control.Contains('schemaVersion = "agent99_outcome_contract_v1"')) "verified outcome contract is present" diff --git a/agent99-control-plane.ps1 b/agent99-control-plane.ps1 index a28b24ff4..ebdd72632 100644 --- a/agent99-control-plane.ps1 +++ b/agent99-control-plane.ps1 @@ -66,6 +66,81 @@ function Get-AgentSshIdentityArguments { @("-i", $SshIdentityFile, "-o", "IdentitiesOnly=yes") } +function Get-AgentSshProcessGuardConfig { + $configured = if ($Config.PSObject.Properties["sshProcessGuard"]) { $Config.sshProcessGuard } else { $null } + [pscustomobject]@{ + enabled = [bool](-not $configured -or -not $configured.PSObject.Properties["enabled"] -or (Convert-AgentBool $configured.enabled)) + staleMinutes = if ($configured -and $configured.PSObject.Properties["staleMinutes"]) { [math]::Max(5, [int]$configured.staleMinutes) } else { 15 } + hardStaleMinutes = if ($configured -and $configured.PSObject.Properties["hardStaleMinutes"]) { [math]::Max(15, [int]$configured.hardStaleMinutes) } else { 60 } + } +} + +function Invoke-AgentStaleSshProcessGuard { + $guard = Get-AgentSshProcessGuardConfig + if (-not $guard.enabled -or -not $ControlledApply) { + return [pscustomobject]@{ ok = $true; skipped = $true; reason = "guard_disabled_or_observe_mode"; stoppedCount = 0; remainingCount = $null; processes = @() } + } + + $targets = @($Config.hosts) + if ($Config.k3s -and $Config.k3s.PSObject.Properties["jumpHost"] -and $Config.k3s.jumpHost) { + $targets += [string]$Config.k3s.jumpHost + } + $targets = @($targets | Select-Object -Unique) + $rows = @() + foreach ($processInfo in @(Get-CimInstance Win32_Process -Filter "Name='ssh.exe'" -ErrorAction SilentlyContinue)) { + try { + $process = Get-Process -Id $processInfo.ProcessId -ErrorAction Stop + $ageMinutes = [math]::Round(((Get-Date) - $process.StartTime).TotalMinutes, 1) + $target = $targets | Where-Object { $processInfo.CommandLine -match [regex]::Escape($_) } | Select-Object -First 1 + $batchMode = [bool]($processInfo.CommandLine -match "BatchMode=yes") + $portForward = [bool]($processInfo.CommandLine -cmatch "(^|\s)-[NLRD](\s|$)") + $parentAlive = [bool](Get-Process -Id $processInfo.ParentProcessId -ErrorAction SilentlyContinue) + $eligible = [bool]( + $ageMinutes -ge $guard.staleMinutes -and + $target -and + $batchMode -and + -not $portForward -and + (-not $parentAlive -or $ageMinutes -ge $guard.hardStaleMinutes) + ) + $stopped = $false + if ($eligible) { + Stop-Process -Id $processInfo.ProcessId -Force -ErrorAction Stop + $stopped = $true + } + $rows += [pscustomobject]@{ + pid = $processInfo.ProcessId + ageMinutes = $ageMinutes + target = $target + batchMode = $batchMode + portForward = $portForward + parentAlive = $parentAlive + eligible = $eligible + stopped = $stopped + } + } catch { + $rows += [pscustomobject]@{ pid = $processInfo.ProcessId; eligible = $false; stopped = $false; error = "process_disappeared_or_stop_failed" } + } + } + + $stoppedCount = @($rows | Where-Object { $_.stopped }).Count + $failures = @($rows | Where-Object { $_.eligible -and -not $_.stopped }).Count + $remainingCount = @(Get-Process ssh -ErrorAction SilentlyContinue).Count + $result = [pscustomobject]@{ + ok = [bool]($failures -eq 0) + skipped = $false + stoppedCount = $stoppedCount + failureCount = $failures + remainingCount = $remainingCount + staleMinutes = $guard.staleMinutes + hardStaleMinutes = $guard.hardStaleMinutes + processes = $rows + } + if ($stoppedCount -gt 0 -or $failures -gt 0) { + Record-AgentEvent "ssh_stale_process_guard" $(if ($failures -gt 0) { "warning" } else { "info" }) "stopped=$stoppedCount failures=$failures remaining=$remainingCount" $result -NoAlert + } + $result +} + function Send-AgentTelegramRelay { param( [string]$Message, @@ -1224,8 +1299,13 @@ function Invoke-SshText { if (-not $process.WaitForExit($TimeoutSeconds * 1000)) { $stopwatch.Stop() - $process.Kill() - $process.WaitForExit(2000) | Out-Null + try { + & taskkill.exe /PID $process.Id /T /F 2>&1 | Out-Null + } catch {} + if (-not $process.WaitForExit(2000) -and -not $process.HasExited) { + $process.Kill() + $process.WaitForExit(2000) | Out-Null + } Remove-Item $stdoutPath, $stderrPath -Force -ErrorAction SilentlyContinue | Out-Null $last = [pscustomobject]@{ ok = $false @@ -3860,6 +3940,7 @@ $backupHealth = $null $providerFreshness = $null $securityTriage = $null $controlTick = $null +$sshProcessGuard = $null $bootState = $null $recoveryTrigger = $null $recoverySlo = $null @@ -3938,6 +4019,10 @@ if ($Mode -in @("Status", "Recover", "Perf")) { $aiServices = Test-AiServices } +if ($Mode -in @("Perf", "LoadShed") -and $ControlledApply) { + $sshProcessGuard = Invoke-AgentStaleSshProcessGuard +} + if ($Mode -in @("Status", "Recover", "Perf", "LoadShed")) { $suppressPerfAlerts = [bool]($ControlledApply -and $Mode -in @("Perf", "LoadShed")) $performance = Get-HostPerformance -SuppressAlerts:$suppressPerfAlerts @@ -4006,6 +4091,7 @@ $result = [pscustomobject]@{ timestamp = (Get-Date -Format o) mode = $Mode controlledApply = [bool]$ControlledApply + sshProcessGuard = $sshProcessGuard bootState = $bootState recoveryTrigger = $recoveryTrigger recoverySlo = $recoverySlo diff --git a/agent99-deploy.ps1 b/agent99-deploy.ps1 index 04259ec69..01c9ed535 100644 --- a/agent99-deploy.ps1 +++ b/agent99-deploy.ps1 @@ -169,6 +169,10 @@ try { $config = Get-Content $ConfigPath -Raw | ConvertFrom-Json Add-DefaultProperty $config "sshIdentityFile" (Join-Path $AgentRoot "keys\agent99_ed25519") + Add-DefaultProperty $config "sshProcessGuard" ([pscustomobject]@{}) + Add-DefaultProperty $config.sshProcessGuard "enabled" $true + Add-DefaultProperty $config.sshProcessGuard "staleMinutes" 15 + Add-DefaultProperty $config.sshProcessGuard "hardStaleMinutes" 60 Add-DefaultProperty $config "k3s" ([pscustomobject]@{}) Add-DefaultProperty $config.k3s "jumpHost" "192.168.0.110" Add-DefaultProperty $config.k3s "preferJumpHost" $true diff --git a/agent99.config.99.example.json b/agent99.config.99.example.json index 5c29501bb..407b64e8b 100644 --- a/agent99.config.99.example.json +++ b/agent99.config.99.example.json @@ -3,6 +3,11 @@ "agentRoot": "C:\\Wooo\\Agent99", "sshUser": "wooo", "sshIdentityFile": "C:\\Wooo\\Agent99\\keys\\agent99_ed25519", + "sshProcessGuard": { + "enabled": true, + "staleMinutes": 15, + "hardStaleMinutes": 60 + }, "sshUsers": { "192.168.0.112": "kali", "192.168.0.188": "ollama" diff --git a/agent99.config.example.json b/agent99.config.example.json index 3855fb95d..9815659e8 100644 --- a/agent99.config.example.json +++ b/agent99.config.example.json @@ -3,6 +3,11 @@ "agentRoot": "C:\\Wooo\\Agent99", "sshUser": "wooo", "sshIdentityFile": "C:\\Wooo\\Agent99\\keys\\agent99_ed25519", + "sshProcessGuard": { + "enabled": true, + "staleMinutes": 15, + "hardStaleMinutes": 60 + }, "sshUsers": {}, "vmrunPath": "", "hosts": [ diff --git a/apps/api/tests/test_agent99_transport_recovery_deploy_contract.py b/apps/api/tests/test_agent99_transport_recovery_deploy_contract.py index 3a7d8b8e6..814f37af4 100644 --- a/apps/api/tests/test_agent99_transport_recovery_deploy_contract.py +++ b/apps/api/tests/test_agent99_transport_recovery_deploy_contract.py @@ -98,3 +98,23 @@ def test_disk_pressure_policy_is_versioned_and_migrated_by_deployer() -> None: assert "$script:ConfigMigrations" in deploy assert '"policyVersion"' in deploy assert "toPolicyVersion = $sourceVersion" in deploy + + +def test_agent99_bounds_stale_ssh_processes_without_touching_tunnels() -> None: + source = CONTROL.read_text(encoding="utf-8") + config = json.loads((ROOT / "agent99.config.99.example.json").read_text()) + guard = source[source.index("function Invoke-AgentStaleSshProcessGuard") :] + guard = guard[: guard.index("function Send-AgentTelegramRelay")] + + assert 'Get-CimInstance Win32_Process -Filter "Name=\'ssh.exe\'"' in guard + assert 'CommandLine -cmatch "(^|\\s)-[NLRD](\\s|$)"' in guard + assert "-not $portForward" in guard + assert 'CommandLine -match "BatchMode=yes"' in guard + assert "Stop-Process -Id $processInfo.ProcessId -Force" in guard + assert "taskkill.exe /PID $process.Id /T /F" in source + assert "-not $process.WaitForExit(2000) -and -not $process.HasExited" in source + assert config["sshProcessGuard"] == { + "enabled": True, + "staleMinutes": 15, + "hardStaleMinutes": 60, + } diff --git a/docs/runbooks/REBOOT-RECOVERY-SOP.md b/docs/runbooks/REBOOT-RECOVERY-SOP.md index 9311e559d..a14e8191b 100644 --- a/docs/runbooks/REBOOT-RECOVERY-SOP.md +++ b/docs/runbooks/REBOOT-RECOVERY-SOP.md @@ -1,9 +1,9 @@ # AWOOOI 重開機恢復 SOP -> **版本**: v5.23 +> **版本**: v5.24 > **最後更新**: 2026-07-10 (台北時間) > **更新者**: Codex -> **觸發事件**: 2026-07-10 全主機重啟後備份/Gitea 可還原性與排程補跑收斂 +> **觸發事件**: 2026-07-10 全主機重啟後 Agent99 transport、效能修復與證據鏈收斂 --- @@ -21,6 +21,18 @@ ## 架構概覽與依賴圖 +### 2026-07-10 P0 Agent99 Transport / Verified Remediation 覆蓋 + +99 的 Agent99 是全主機恢復控制節點。Windows 開機後由既有 Agent99 scheduled tasks 啟動；`Status` 讀到 boot identity 變更時，只能用 single-flight queue 觸發一次 `Recover`，並以 10 分鐘 cooldown 避免併發恢復。每次執行必須寫入 `bootState`、`recoveryTrigger` 與 `recoverySlo` evidence。只有實際全主機重啟事件、五主機 reachability、必要服務、public routes 與 freshness 全部在同一個 10 分鐘 window 通過，才可宣稱 reboot recovery SLO 達標。 + +Agent99 SSH transport 使用 99 本機專用 Ed25519 identity `C:\Wooo\Agent99\keys\agent99_ed25519`；private key 不得輸出、複製到 evidence、Telegram 或 repo。只允許將 public key 以 `from="192.168.0.99",restrict` 授權到 110 `wooo` 與 112 `kali`。固定 route map 為 110 / 112 direct，120 / 121 / 188 優先經 110 jump；禁止以共用密碼或任意 interactive shell 取代。Runtime 必須由 `agent99-deploy.ps1` 經 parse、contract、synthetic、staging、backup、atomic promote 與 rollback gate 部署，`runtime-manifest.json.sourceRevision` 必須等於 Gitea commit SHA。 + +所有 SSH command 都必須 bounded timeout。Timeout 時先用 `taskkill /T` 終止 process tree，再確認 process 是否仍存活；`Perf` / `LoadShed` controlled apply 執行前會清理只符合以下全部條件的 stale client：Agent99 allowlisted target、`BatchMode=yes`、沒有 `-N/-L/-R/-D` tunnel、超過 15 分鐘且 parent 已消失，或超過 60 分鐘 hard limit。guard 不記錄 command line，不碰互動式 SSH 或 tunnel，結果寫入 `sshProcessGuard` evidence。 + +效能修復不得以 command exit code 取代 post-verifier。只要 action 配置 verifier，成功判定完全由 verifier 決定；verifier 失敗必須記為 failed，僅建立 10 分鐘 failure backoff，不得建立 success cooldown。policy 變更必須提升 `policyVersion`，由 deployer migration 更新 99 live config，避免舊 command 或 marker 繼續造成 false green。 + +本輪 live receipt：`agent99-load-shed-marker-correction.json` 已封存 188 舊 false-success marker，`agent99-stale-ssh-cleanup-v2.json` 已受控終止 7 個逾時 orphan SSH client；`agent99-Perf-20260710-224454.json` 顯示 110、112、120、121、188 全部 readback OK，route 分別為 direct、direct、via110、via110、via110。110 load/core `0.60`、disk `84%`；188 load/core `0.18`、disk 由 `90%` 降至 `88%`。這證明 transport、效能讀回與 verifier 修復 lane 已恢復，不等於已完成下一次實際全主機 cold-start 的 10 分鐘 SLA 驗收。 + ### 2026-07-10 P0 備份/Gitea 自動驗證覆蓋 每次 host reboot 後，備份 lane 必須依序執行 `/backup/scripts/backup-status.sh --no-notify`、`/backup/scripts/verify-gitea-backup.sh --no-notify`，再由 110 的每日排程執行 `/backup/scripts/gitea-bundle-backup-sync-188.sh`。最後讀回 110/188 `backup_health.prom`、Gitea restore status、bundle sync status、188 sample restore metric 與 Prometheus Backup/Gitea alerts；不得只看檔案 mtime、HTTP 200 或命令 exit code。