fix(agent99): bound stale transport processes

AGENT99-AI-WORKSTATION-SOP.md

@@ -709,3 +709,38 @@ Agent99 now accepts operator text instructions and turns them into allowlisted q
   - Evidence: `C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-224134.json`.
   - Public smoke: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
   - Telegram result: `operator_command_result`, `severity=info`, `sent=true`, `relayOk=true`.
+
+## 2026-07-10 M9 Source-Trusted Transport And Verified Remediation Receipt
+
+Agent99 now uses a source-trusted deployment path, a dedicated bounded SSH transport, and verifier-authoritative remediation outcomes.
+
+- Source authority and deployment:
+  - Gitea commit SHA is the source revision; a local scratch checkout or a Windows file timestamp is not deployment truth.
+  - `agent99-deploy.ps1` validates PowerShell parse, Agent99 contracts, synthetic checks, staging content, backup, atomic promotion, task registration, and rollback.
+  - `C:\Wooo\Agent99\runtime-manifest.json.sourceRevision` must match the deployed Gitea SHA. A successful source test without this runtime readback is not a completed deployment.
+- Dedicated transport:
+  - Private identity: `C:\Wooo\Agent99\keys\agent99_ed25519`; it remains on 99, is ACL-restricted, and is never copied into evidence, Telegram, logs, or the repository.
+  - Public-key authorization is limited with `from="192.168.0.99",restrict` on 110 user `wooo` and 112 user `kali`.
+  - Route map: 110 and 112 direct; 120, 121, and 188 prefer 110 as the jump host.
+  - Live public-key fingerprint receipt: `SHA256:4ccZZSe0buoWyivi2mfTMXE9LZfCmFBPr/evCzZ0nzA`.
+- Automatic recovery:
+  - Agent99 persists boot identity in `C:\Wooo\Agent99\state\boot-state.json`.
+  - A changed boot observation queues one Recover request only; duplicate requests are suppressed by single-flight state and a 10-minute cooldown.
+  - `recovery_slo_result` records the recovery window. It may claim success only after a real reboot drill proves all required hosts, services, routes, and freshness checks inside 10 minutes.
+- Verifier-authoritative remediation:
+  - When an action configures `postVerify`, command success alone is insufficient. The verifier result is the effective action result.
+  - A failed verifier creates only the 10-minute failure backoff; it cannot create a success cooldown.
+  - `policyVersion` migration updates stale live policies and prevents an old command or marker from surviving a new source deployment.
+- Bounded SSH process lifecycle:
+  - Every SSH call has a timeout; timeout cleanup terminates the SSH process tree and verifies that the client exited.
+  - Before controlled `Perf` or `LoadShed`, the stale-process guard only considers Agent99 allowlisted targets using `BatchMode=yes` and explicitly excludes `-N`, `-L`, `-R`, and `-D` tunnels.
+  - A process is eligible after 15 minutes with a missing parent, or after the 60-minute hard limit. Evidence records selected process metadata but never the command line.
+- Live receipts:
+  - Key provisioning: `C:\Wooo\Agent99\evidence\agent99-ssh-key-provision-final.json`.
+  - False-success marker correction: `C:\Wooo\Agent99\evidence\agent99-load-shed-marker-correction.json`.
+  - Controlled stale-client cleanup: `C:\Wooo\Agent99\evidence\agent99-stale-ssh-cleanup-v2.json`; 7 orphan clients stopped.
+  - Performance readback: `C:\Wooo\Agent99\evidence\agent99-Perf-20260710-224454.json`; all five hosts OK, 188 disk reduced from 90% to 88%.
+  - The deployment receipt and runtime manifest must contain the final deployed Gitea SHA; paths or source-test success alone are not accepted as runtime proof.
+- Remaining acceptance truth:
+  - This receipt closes the transport stall, orphan SSH, 188 disk pressure, and false-green remediation defects.
+  - It does not close the whole-host 10-minute cold-start objective until the next controlled reboot produces a fresh scorecard with all required gates green.

agent99-contract-check.ps1

@@ -66,6 +66,7 @@ $bootstrap = [IO.File]::ReadAllText((Join-Path $SourceRoot "agent99-bootstrap.ps
 
 Add-Check "transport:dedicated_identity" ($control.Contains("sshIdentityFile") -and $control.Contains("IdentitiesOnly=yes")) "dedicated SSH identity is enforced"
 Add-Check "transport:jump_first" ($control.Contains("preferJumpHost") -and $control.Contains("ssh_jump_fallback")) "preferred jump with bounded fallback is present"
+Add-Check "transport:stale_process_guard" ($control.Contains("Invoke-AgentStaleSshProcessGuard") -and $control.Contains("taskkill.exe /PID") -and $control.Contains("ssh_stale_process_guard")) "timed-out and stale SSH clients are bounded"
 Add-Check "recovery:auto_trigger" ($control.Contains("Get-AgentBootState") -and $control.Contains("Start-AgentRecoveryFromObservation") -and $control.Contains("recovery_already_queued")) "boot and host-down recovery use a single-flight queue"
 Add-Check "recovery:slo" ($control.Contains("recovery_slo_result") -and $control.Contains("withinSlo")) "10-minute recovery evidence is present"
 Add-Check "outcome:verified" ($control.Contains("Get-AgentOutcomeContract") -and $control.Contains('schemaVersion = "agent99_outcome_contract_v1"')) "verified outcome contract is present"

agent99-control-plane.ps1

@@ -66,6 +66,81 @@ function Get-AgentSshIdentityArguments {
   @("-i", $SshIdentityFile, "-o", "IdentitiesOnly=yes")
 }
 
+function Get-AgentSshProcessGuardConfig {
+  $configured = if ($Config.PSObject.Properties["sshProcessGuard"]) { $Config.sshProcessGuard } else { $null }
+  [pscustomobject]@{
+    enabled = [bool](-not $configured -or -not $configured.PSObject.Properties["enabled"] -or (Convert-AgentBool $configured.enabled))
+    staleMinutes = if ($configured -and $configured.PSObject.Properties["staleMinutes"]) { [math]::Max(5, [int]$configured.staleMinutes) } else { 15 }
+    hardStaleMinutes = if ($configured -and $configured.PSObject.Properties["hardStaleMinutes"]) { [math]::Max(15, [int]$configured.hardStaleMinutes) } else { 60 }
+  }
+}
+
+function Invoke-AgentStaleSshProcessGuard {
+  $guard = Get-AgentSshProcessGuardConfig
+  if (-not $guard.enabled -or -not $ControlledApply) {
+    return [pscustomobject]@{ ok = $true; skipped = $true; reason = "guard_disabled_or_observe_mode"; stoppedCount = 0; remainingCount = $null; processes = @() }
+  }
+
+  $targets = @($Config.hosts)
+  if ($Config.k3s -and $Config.k3s.PSObject.Properties["jumpHost"] -and $Config.k3s.jumpHost) {
+    $targets += [string]$Config.k3s.jumpHost
+  }
+  $targets = @($targets | Select-Object -Unique)
+  $rows = @()
+  foreach ($processInfo in @(Get-CimInstance Win32_Process -Filter "Name='ssh.exe'" -ErrorAction SilentlyContinue)) {
+    try {
+      $process = Get-Process -Id $processInfo.ProcessId -ErrorAction Stop
+      $ageMinutes = [math]::Round(((Get-Date) - $process.StartTime).TotalMinutes, 1)
+      $target = $targets | Where-Object { $processInfo.CommandLine -match [regex]::Escape($_) } | Select-Object -First 1
+      $batchMode = [bool]($processInfo.CommandLine -match "BatchMode=yes")
+      $portForward = [bool]($processInfo.CommandLine -cmatch "(^|\s)-[NLRD](\s|$)")
+      $parentAlive = [bool](Get-Process -Id $processInfo.ParentProcessId -ErrorAction SilentlyContinue)
+      $eligible = [bool](
+        $ageMinutes -ge $guard.staleMinutes -and
+        $target -and
+        $batchMode -and
+        -not $portForward -and
+        (-not $parentAlive -or $ageMinutes -ge $guard.hardStaleMinutes)
+      )
+      $stopped = $false
+      if ($eligible) {
+        Stop-Process -Id $processInfo.ProcessId -Force -ErrorAction Stop
+        $stopped = $true
+      }
+      $rows += [pscustomobject]@{
+        pid = $processInfo.ProcessId
+        ageMinutes = $ageMinutes
+        target = $target
+        batchMode = $batchMode
+        portForward = $portForward
+        parentAlive = $parentAlive
+        eligible = $eligible
+        stopped = $stopped
+      }
+    } catch {
+      $rows += [pscustomobject]@{ pid = $processInfo.ProcessId; eligible = $false; stopped = $false; error = "process_disappeared_or_stop_failed" }
+    }
+  }
+
+  $stoppedCount = @($rows | Where-Object { $_.stopped }).Count
+  $failures = @($rows | Where-Object { $_.eligible -and -not $_.stopped }).Count
+  $remainingCount = @(Get-Process ssh -ErrorAction SilentlyContinue).Count
+  $result = [pscustomobject]@{
+    ok = [bool]($failures -eq 0)
+    skipped = $false
+    stoppedCount = $stoppedCount
+    failureCount = $failures
+    remainingCount = $remainingCount
+    staleMinutes = $guard.staleMinutes
+    hardStaleMinutes = $guard.hardStaleMinutes
+    processes = $rows
+  }
+  if ($stoppedCount -gt 0 -or $failures -gt 0) {
+    Record-AgentEvent "ssh_stale_process_guard" $(if ($failures -gt 0) { "warning" } else { "info" }) "stopped=$stoppedCount failures=$failures remaining=$remainingCount" $result -NoAlert
+  }
+  $result
+}
+
 function Send-AgentTelegramRelay {
   param(
     [string]$Message,
@@ -1224,8 +1299,13 @@ function Invoke-SshText {
 
       if (-not $process.WaitForExit($TimeoutSeconds * 1000)) {
         $stopwatch.Stop()
-        $process.Kill()
-        $process.WaitForExit(2000) | Out-Null
+        try {
+          & taskkill.exe /PID $process.Id /T /F 2>&1 | Out-Null
+        } catch {}
+        if (-not $process.WaitForExit(2000) -and -not $process.HasExited) {
+          $process.Kill()
+          $process.WaitForExit(2000) | Out-Null
+        }
         Remove-Item $stdoutPath, $stderrPath -Force -ErrorAction SilentlyContinue | Out-Null
         $last = [pscustomobject]@{
           ok = $false
@@ -3860,6 +3940,7 @@ $backupHealth = $null
 $providerFreshness = $null
 $securityTriage = $null
 $controlTick = $null
+$sshProcessGuard = $null
 $bootState = $null
 $recoveryTrigger = $null
 $recoverySlo = $null
@@ -3938,6 +4019,10 @@ if ($Mode -in @("Status", "Recover", "Perf")) {
   $aiServices = Test-AiServices
 }
 
+if ($Mode -in @("Perf", "LoadShed") -and $ControlledApply) {
+  $sshProcessGuard = Invoke-AgentStaleSshProcessGuard
+}
+
 if ($Mode -in @("Status", "Recover", "Perf", "LoadShed")) {
   $suppressPerfAlerts = [bool]($ControlledApply -and $Mode -in @("Perf", "LoadShed"))
   $performance = Get-HostPerformance -SuppressAlerts:$suppressPerfAlerts
@@ -4006,6 +4091,7 @@ $result = [pscustomobject]@{
   timestamp = (Get-Date -Format o)
   mode = $Mode
   controlledApply = [bool]$ControlledApply
+  sshProcessGuard = $sshProcessGuard
   bootState = $bootState
   recoveryTrigger = $recoveryTrigger
   recoverySlo = $recoverySlo

agent99-deploy.ps1

@@ -169,6 +169,10 @@ try {
 
   $config = Get-Content $ConfigPath -Raw | ConvertFrom-Json
   Add-DefaultProperty $config "sshIdentityFile" (Join-Path $AgentRoot "keys\agent99_ed25519")
+  Add-DefaultProperty $config "sshProcessGuard" ([pscustomobject]@{})
+  Add-DefaultProperty $config.sshProcessGuard "enabled" $true
+  Add-DefaultProperty $config.sshProcessGuard "staleMinutes" 15
+  Add-DefaultProperty $config.sshProcessGuard "hardStaleMinutes" 60
   Add-DefaultProperty $config "k3s" ([pscustomobject]@{})
   Add-DefaultProperty $config.k3s "jumpHost" "192.168.0.110"
   Add-DefaultProperty $config.k3s "preferJumpHost" $true

agent99.config.99.example.json

@@ -3,6 +3,11 @@
   "agentRoot": "C:\\Wooo\\Agent99",
   "sshUser": "wooo",
   "sshIdentityFile": "C:\\Wooo\\Agent99\\keys\\agent99_ed25519",
+  "sshProcessGuard": {
+    "enabled": true,
+    "staleMinutes": 15,
+    "hardStaleMinutes": 60
+  },
   "sshUsers": {
     "192.168.0.112": "kali",
     "192.168.0.188": "ollama"

agent99.config.example.json

@@ -3,6 +3,11 @@
   "agentRoot": "C:\\Wooo\\Agent99",
   "sshUser": "wooo",
   "sshIdentityFile": "C:\\Wooo\\Agent99\\keys\\agent99_ed25519",
+  "sshProcessGuard": {
+    "enabled": true,
+    "staleMinutes": 15,
+    "hardStaleMinutes": 60
+  },
   "sshUsers": {},
   "vmrunPath": "",
   "hosts": [

apps/api/tests/test_agent99_transport_recovery_deploy_contract.py

@@ -98,3 +98,23 @@ def test_disk_pressure_policy_is_versioned_and_migrated_by_deployer() -> None:
     assert "$script:ConfigMigrations" in deploy
     assert '"policyVersion"' in deploy
     assert "toPolicyVersion = $sourceVersion" in deploy
+
+
+def test_agent99_bounds_stale_ssh_processes_without_touching_tunnels() -> None:
+    source = CONTROL.read_text(encoding="utf-8")
+    config = json.loads((ROOT / "agent99.config.99.example.json").read_text())
+    guard = source[source.index("function Invoke-AgentStaleSshProcessGuard") :]
+    guard = guard[: guard.index("function Send-AgentTelegramRelay")]
+
+    assert 'Get-CimInstance Win32_Process -Filter "Name=\'ssh.exe\'"' in guard
+    assert 'CommandLine -cmatch "(^|\\s)-[NLRD](\\s|$)"' in guard
+    assert "-not $portForward" in guard
+    assert 'CommandLine -match "BatchMode=yes"' in guard
+    assert "Stop-Process -Id $processInfo.ProcessId -Force" in guard
+    assert "taskkill.exe /PID $process.Id /T /F" in source
+    assert "-not $process.WaitForExit(2000) -and -not $process.HasExited" in source
+    assert config["sshProcessGuard"] == {
+        "enabled": True,
+        "staleMinutes": 15,
+        "hardStaleMinutes": 60,
+    }

docs/runbooks/REBOOT-RECOVERY-SOP.md

@@ -1,9 +1,9 @@
 # AWOOOI 重開機恢復 SOP
 
-> **版本**: v5.23
+> **版本**: v5.24
 > **最後更新**: 2026-07-10 (台北時間)
 > **更新者**: Codex
-> **觸發事件**: 2026-07-10 全主機重啟後備份/Gitea 可還原性與排程補跑收斂
+> **觸發事件**: 2026-07-10 全主機重啟後 Agent99 transport、效能修復與證據鏈收斂
 
 ---
 
@@ -21,6 +21,18 @@
 
 ## 架構概覽與依賴圖
 
+### 2026-07-10 P0 Agent99 Transport / Verified Remediation 覆蓋
+
+99 的 Agent99 是全主機恢復控制節點。Windows 開機後由既有 Agent99 scheduled tasks 啟動；`Status` 讀到 boot identity 變更時，只能用 single-flight queue 觸發一次 `Recover`，並以 10 分鐘 cooldown 避免併發恢復。每次執行必須寫入 `bootState`、`recoveryTrigger` 與 `recoverySlo` evidence。只有實際全主機重啟事件、五主機 reachability、必要服務、public routes 與 freshness 全部在同一個 10 分鐘 window 通過，才可宣稱 reboot recovery SLO 達標。
+
+Agent99 SSH transport 使用 99 本機專用 Ed25519 identity `C:\Wooo\Agent99\keys\agent99_ed25519`；private key 不得輸出、複製到 evidence、Telegram 或 repo。只允許將 public key 以 `from="192.168.0.99",restrict` 授權到 110 `wooo` 與 112 `kali`。固定 route map 為 110 / 112 direct，120 / 121 / 188 優先經 110 jump；禁止以共用密碼或任意 interactive shell 取代。Runtime 必須由 `agent99-deploy.ps1` 經 parse、contract、synthetic、staging、backup、atomic promote 與 rollback gate 部署，`runtime-manifest.json.sourceRevision` 必須等於 Gitea commit SHA。
+
+所有 SSH command 都必須 bounded timeout。Timeout 時先用 `taskkill /T` 終止 process tree，再確認 process 是否仍存活；`Perf` / `LoadShed` controlled apply 執行前會清理只符合以下全部條件的 stale client：Agent99 allowlisted target、`BatchMode=yes`、沒有 `-N/-L/-R/-D` tunnel、超過 15 分鐘且 parent 已消失，或超過 60 分鐘 hard limit。guard 不記錄 command line，不碰互動式 SSH 或 tunnel，結果寫入 `sshProcessGuard` evidence。
+
+效能修復不得以 command exit code 取代 post-verifier。只要 action 配置 verifier，成功判定完全由 verifier 決定；verifier 失敗必須記為 failed，僅建立 10 分鐘 failure backoff，不得建立 success cooldown。policy 變更必須提升 `policyVersion`，由 deployer migration 更新 99 live config，避免舊 command 或 marker 繼續造成 false green。
+
+本輪 live receipt：`agent99-load-shed-marker-correction.json` 已封存 188 舊 false-success marker，`agent99-stale-ssh-cleanup-v2.json` 已受控終止 7 個逾時 orphan SSH client；`agent99-Perf-20260710-224454.json` 顯示 110、112、120、121、188 全部 readback OK，route 分別為 direct、direct、via110、via110、via110。110 load/core `0.60`、disk `84%`；188 load/core `0.18`、disk 由 `90%` 降至 `88%`。這證明 transport、效能讀回與 verifier 修復 lane 已恢復，不等於已完成下一次實際全主機 cold-start 的 10 分鐘 SLA 驗收。
+
 ### 2026-07-10 P0 備份/Gitea 自動驗證覆蓋
 
 每次 host reboot 後，備份 lane 必須依序執行 `/backup/scripts/backup-status.sh --no-notify`、`/backup/scripts/verify-gitea-backup.sh --no-notify`，再由 110 的每日排程執行 `/backup/scripts/gitea-bundle-backup-sync-188.sh`。最後讀回 110/188 `backup_health.prom`、Gitea restore status、bundle sync status、188 sample restore metric 與 Prometheus Backup/Gitea alerts；不得只看檔案 mtime、HTTP 200 或命令 exit code。