From e8b0244ee7b8dcd2dbae4b86826d154894814266 Mon Sep 17 00:00:00 2001 From: ogt Date: Sat, 11 Jul 2026 20:45:52 +0800 Subject: [PATCH] feat(agent99): add durable completion callback --- AGENT99-AI-WORKSTATION-SOP.md | 13 +- HOST-REBOOT-AI-AUTOMATION-SOP.md | 5 + agent99-contract-check.ps1 | 11 +- agent99-control-plane.ps1 | 229 ++++++++++++++- agent99-deploy.ps1 | 8 + agent99.config.99.example.json | 9 + agent99.config.example.json | 9 + apps/api/src/api/v1/agents.py | 58 +++- apps/api/src/models/agent99_completion.py | 73 +++++ .../services/agent99_completion_callback.py | 219 ++++++++++++++ .../test_agent99_completion_callback_api.py | 270 ++++++++++++++++++ ...enterprise_ai_automation_work_items_api.py | 14 +- ...ise-ai-automation-work-items.snapshot.json | 69 +++-- docs/runbooks/REBOOT-RECOVERY-SOP.md | 2 + ...99-enterprise-ai-automation-master-plan.md | 17 +- 15 files changed, 965 insertions(+), 41 deletions(-) create mode 100644 apps/api/src/models/agent99_completion.py create mode 100644 apps/api/src/services/agent99_completion_callback.py create mode 100644 apps/api/tests/test_agent99_completion_callback_api.py diff --git a/AGENT99-AI-WORKSTATION-SOP.md b/AGENT99-AI-WORKSTATION-SOP.md index 4c9b29b68..7ee5d417c 100644 --- a/AGENT99-AI-WORKSTATION-SOP.md +++ b/AGENT99-AI-WORKSTATION-SOP.md @@ -757,7 +757,18 @@ Agent99 now uses a source-trusted deployment path, a dedicated bounded SSH trans - False-success marker correction: `C:\Wooo\Agent99\evidence\agent99-load-shed-marker-correction.json`. - Controlled stale-client cleanup: `C:\Wooo\Agent99\evidence\agent99-stale-ssh-cleanup-v2.json`; 7 orphan clients stopped. - Performance readback: `C:\Wooo\Agent99\evidence\agent99-Perf-20260710-224454.json`; all five hosts OK, 188 disk reduced from 90% to 88%. - - The deployment receipt and runtime manifest must contain the final deployed Gitea SHA; paths or source-test success alone are not accepted as runtime proof. +- The deployment receipt and runtime manifest must contain the final deployed Gitea SHA; paths or source-test success alone are not accepted as runtime proof. - Remaining acceptance truth: - This receipt closes the transport stall, orphan SSH, 188 disk pressure, and false-green remediation defects. - It does not close the whole-host 10-minute cold-start objective until the next controlled reboot produces a fresh scorecard with all required gates green. + +## 2026-07-11 Runtime Truth And Completion Callback Contract + +- `scripts/reboot-recovery/agent99-live-preflight.ps1` now treats a required evidence file as usable only when it is fresh, JSON-parseable and content-healthy. A fresh timestamp with a failed safe-summary adapter is a blocker, not green. +- SRE and Telegram inbox evidence use bounded shape adapters. The preflight reads counts, status and exit code only; it never reads raw Telegram updates, message text, token, chat id, `.env`, auth data, sessions or SQLite. +- A successful `Recover` emits one `recovery_slo_result` lifecycle event even at `info` severity. The visual is a recovery timeline with scope, duration, run id and all seven phase verifiers; `agent_complete` remains no-alert to prevent duplicate noise. +- Same-state Telegram suppression must append a `dedupe_window` attempt receipt. Silence without a receipt is invalid. +- Production verifier: run `Wooo-Agent99-Startup-Recovery` under the registered `WOOO` S4U principal, then pipe `scripts/reboot-recovery/agent99-recover-receipt-readback.ps1` to Windows PowerShell. Closure requires `RECOVERY_VERIFIED=1` and `NOTIFICATION_RECEIPTED=1`; Administrator launcher output is not a substitute. +- Canonical completion is not Telegram delivery. Agent99 posts a bounded `agent99_completion_callback_v1` payload authenticated by the existing `AGENT99_SRE_RELAY_TOKEN`; raw logs, secrets and full evidence paths are rejected. +- The API writes an idempotent Agent99 channel event, AWOOOI `EXECUTION_COMPLETED`/verified `RESOLVED` operation receipts and an AwoooP completed shadow run. Agent99 moves a callback from `state\completion-callbacks\pending` to `processed` only after the API returns the same callback/run/trace and `durable_readback=true` with operation receipts. +- Callback failures retry in bounded batches after cooldown and never repeat the repair action. The callback source is not production-complete until Gitea CD, API runtime readback, Windows client deploy and one same-trace SRE replay all pass. diff --git a/HOST-REBOOT-AI-AUTOMATION-SOP.md b/HOST-REBOOT-AI-AUTOMATION-SOP.md index ddba2f27c..417b3fd90 100644 --- a/HOST-REBOOT-AI-AUTOMATION-SOP.md +++ b/HOST-REBOOT-AI-AUTOMATION-SOP.md @@ -213,6 +213,11 @@ Actions still requiring explicit break-glass: Recovery is complete only when all of the following are captured: +- Agent99 live preflight shows runtime manifest matched, tasks `9/9`, relay listening, active queues clear and every required evidence row `fresh=true`, `parsed=true`, `contentHealthy=true`, `usable=true`. +- The production-principal Recover receipt contains all seven phases: `vm_start`, `host_reachability`, `host112_guest_readiness`, `harbor_registry`, `k3s_workloads`, `public_routes`, and `full_sop_scorecard`. +- Telegram has one lifecycle receipt for the run: either a real `sent + visualSent` recovery timeline or an explicit same-state `dedupe_window` suppression backed by a previous sent visual receipt. +- For alert-triggered recovery, the identical callback/run/trace is durably readable from AWOOOI operation history and the AwoooP run projection. Telegram success alone never closes the incident. + - Five VM hosts have reachability evidence. - 110 Harbor core is healthy. - 120 AWOOOI deployments are ready. diff --git a/agent99-contract-check.ps1 b/agent99-contract-check.ps1 index e37427de6..f63fea863 100644 --- a/agent99-contract-check.ps1 +++ b/agent99-contract-check.ps1 @@ -58,8 +58,14 @@ foreach ($name in @("agent99.config.example.json", "agent99.config.99.example.js $config.recoveryCoordinator.controllerHost -eq "192.168.0.110" -and @($config.recoveryCoordinator.requiredHostAliases).Count -eq 7 ) - $ok = $hasHosts -and $hasRoutes -and $hasSlo -and $hasIdentity -and $hasJump -and $hasAutoRecovery -and $hasRecoveryCoordinator - Add-Check "config:$name" $ok "hosts=$(@($config.hosts).Count) routes=$(@($config.publicUrls).Count) slo10m=$hasSlo identity=$hasIdentity jump=$hasJump autoRecovery=$hasAutoRecovery recoveryCoordinator=$hasRecoveryCoordinator" + $hasCompletionCallback = [bool]( + $config.completionCallback -and + $config.completionCallback.enabled -eq $true -and + $config.completionCallback.url -match "^https://" -and + $config.completionCallback.tokenEnv -eq "AGENT99_SRE_RELAY_TOKEN" + ) + $ok = $hasHosts -and $hasRoutes -and $hasSlo -and $hasIdentity -and $hasJump -and $hasAutoRecovery -and $hasRecoveryCoordinator -and $hasCompletionCallback + Add-Check "config:$name" $ok "hosts=$(@($config.hosts).Count) routes=$(@($config.publicUrls).Count) slo10m=$hasSlo identity=$hasIdentity jump=$hasJump autoRecovery=$hasAutoRecovery recoveryCoordinator=$hasRecoveryCoordinator completionCallback=$hasCompletionCallback" } catch { Add-Check "config:$name" $false "json_parse_failed: $($_.Exception.Message)" } @@ -82,6 +88,7 @@ Add-Check "recovery:slo" ($control.Contains("recovery_slo_result") -and $control Add-Check "recovery:full_sop_coordinator" ($control.Contains("Invoke-AgentRecoveryCoordinatorReadback") -and $control.Contains("full_sop_scorecard") -and $control.Contains("SCORECARD_FRESH_REBOOT_WINDOW") -and $control.Contains('scope = $recoveryScope')) "Agent99 cannot close reboot recovery without the 110 full-SOP scorecard" Add-Check "outcome:verified" ($control.Contains("Get-AgentOutcomeContract") -and $control.Contains('schemaVersion = "agent99_outcome_contract_v1"')) "verified outcome contract is present" Add-Check "problem_counts:v2" ($control.Contains('schemaVersion = "agent99_problem_counts_v2"') -and $control.Contains("totalVerifiedResolved")) "verified problem counts are present" +Add-Check "callback:durable_completion" ($control.Contains('schema_version = "agent99_completion_callback_v1"') -and $control.Contains("durable_readback") -and $control.Contains("completion-callbacks\pending")) "Agent99 completion callbacks are queued until durable AWOOOI/AwoooP readback" Add-Check "runtime:manifest" ($control.Contains("Test-AgentRuntimeManifest") -and $bootstrap.Contains('schemaVersion = "agent99_runtime_manifest_v1"')) "runtime source drift is self-checked" Add-Check "routing:structured" ($inbox.Contains("Resolve-AgentAlertRoute") -and $inbox.Contains('schemaVersion = "agent99_sre_alert_single_flight_v1"')) "structured route and local single-flight are present" Add-Check "routing:telegram" ($telegram.Contains('schemaVersion = "agent99_alert_route_v1"') -and $telegram.Contains("autoIngestMonitoringAlerts")) "Telegram structured ingress is present" diff --git a/agent99-control-plane.ps1 b/agent99-control-plane.ps1 index a79d77d30..0b8f79547 100644 --- a/agent99-control-plane.ps1 +++ b/agent99-control-plane.ps1 @@ -4173,6 +4173,226 @@ function Update-AgentProblemKnowledge { } } +function Get-AgentCompletionCallbackSettings { + if (-not $Config.PSObject.Properties["completionCallback"] -or -not $Config.completionCallback) { + return [pscustomobject]@{ enabled = $false } + } + $settings = $Config.completionCallback + [pscustomobject]@{ + enabled = Convert-AgentBool (Get-AgentObjectValue $settings "enabled" $false) + url = [string](Get-AgentObjectValue $settings "url" "") + tokenEnv = [string](Get-AgentObjectValue $settings "tokenEnv" "AGENT99_SRE_RELAY_TOKEN") + timeoutSeconds = [math]::Max(3, [math]::Min(30, [int](Get-AgentObjectValue $settings "timeoutSeconds" 15))) + maxAttempts = [math]::Max(1, [math]::Min(3, [int](Get-AgentObjectValue $settings "maxAttempts" 2))) + batchLimit = [math]::Max(1, [math]::Min(10, [int](Get-AgentObjectValue $settings "batchLimit" 5))) + retryAfterMinutes = [math]::Max(1, [math]::Min(60, [int](Get-AgentObjectValue $settings "retryAfterMinutes" 5))) + } +} + +function Get-AgentEnvironmentValue { + param([string]$Name) + if (-not $Name) { return $null } + foreach ($scope in @("Machine", "User", "Process")) { + $value = [Environment]::GetEnvironmentVariable($Name, $scope) + if ($value) { return [string]$value } + } + return $null +} + +function Invoke-AgentCompletionCallback { + param( + [object]$Result = $null, + [object]$TelegramAttempt = $null, + [string]$PayloadPath = "" + ) + + $settings = Get-AgentCompletionCallbackSettings + if (-not $settings.enabled) { + return [pscustomobject]@{ ok = $false; status = "disabled"; durableReadback = $false } + } + + $callbackRoot = Join-Path $Config.agentRoot "state\completion-callbacks" + $pendingDir = Join-Path $callbackRoot "pending" + $processedDir = Join-Path $callbackRoot "processed" + New-Item -ItemType Directory -Force -Path $pendingDir, $processedDir | Out-Null + + $payload = $null + $pendingPath = $PayloadPath + if ($PayloadPath) { + try { + $payload = Get-Content -Path $PayloadPath -Raw | ConvertFrom-Json + } catch { + return [pscustomobject]@{ + ok = $false + status = "pending_payload_invalid" + durableReadback = $false + errorType = $_.Exception.GetType().Name + } + } + } else { + $outcome = if ($Result -and $Result.PSObject.Properties["outcome"]) { $Result.outcome } else { $null } + $outcomeState = [string](Get-AgentObjectValue $outcome "state" (Get-AgentObjectValue $Result "outcomeState" "failed")) + $resultId = [string](Get-AgentObjectValue $Result "id" "unknown") + $alertId = [string](Get-AgentObjectValue $Result "alertId" "") + $correlationKey = [string](Get-AgentObjectValue $Result "correlationKey" "") + $callbackId = Convert-AgentSafeKey "agent99:$resultId`:$outcomeState" + $evidencePath = [string](Get-AgentObjectValue $Result "evidence" "") + $visualPath = [string](Get-AgentObjectValue $TelegramAttempt "visualPath" "") + $problem = Get-AgentObjectValue $Result "problem" $null + $payload = [ordered]@{ + schema_version = "agent99_completion_callback_v1" + callback_id = $callbackId + project_id = "awoooi" + run_id = $resultId + trace_id = if ($correlationKey) { $correlationKey } else { $resultId } + work_item_id = if ($alertId) { "agent99-incident:$alertId" } else { "agent99-command:$resultId" } + alert_id = if ($alertId) { $alertId } else { $null } + correlation_key = if ($correlationKey) { $correlationKey } else { $null } + source = [string](Get-AgentObjectValue $Result "source" "agent99") + mode = [string](Get-AgentObjectValue $Result "mode" "Status") + outcome_state = $outcomeState + controlled_apply = Convert-AgentBool (Get-AgentObjectValue $Result "controlledApply" $false) + transport_ok = Convert-AgentBool (Get-AgentObjectValue $Result "transportOk" $false) + verifier_name = [string](Get-AgentObjectValue $outcome "verifierName" "agent99_mode_post_condition") + verifier_passed = Convert-AgentBool (Get-AgentObjectValue $outcome "verifierPassed" $false) + source_event_resolved = Convert-AgentBool (Get-AgentObjectValue $outcome "sourceEventResolved" $false) + source_event_resolution_policy = [string](Get-AgentObjectValue $Result "sourceEventResolutionPolicy" "external_source_receipt_required") + duration_seconds = [double](Get-AgentObjectValue $Result "durationSeconds" 0) + evidence_ref = if ($evidencePath) { Split-Path -Leaf $evidencePath } else { "evidence-missing.json" } + alert_kind = [string](Get-AgentObjectValue $Result "alertKind" "") + alert_service = [string](Get-AgentObjectValue $Result "alertService" "") + alert_host = [string](Get-AgentObjectValue $Result "alertHost" "") + telegram = [ordered]@{ + sent = Convert-AgentBool (Get-AgentObjectValue $TelegramAttempt "sent" $false) + suppressed = Convert-AgentBool (Get-AgentObjectValue $TelegramAttempt "suppressed" $false) + reason = [string](Get-AgentObjectValue $TelegramAttempt "reason" "") + message_format = [string](Get-AgentObjectValue $TelegramAttempt "messageFormat" "") + visual_sent = Convert-AgentBool (Get-AgentObjectValue $TelegramAttempt "visualSent" $false) + visual_file = if ($visualPath) { Split-Path -Leaf $visualPath } else { $null } + } + problem = [ordered]@{ + key = [string](Get-AgentObjectValue $problem "key" "") + occurrences = [int](Get-AgentObjectValue $problem "occurrences" 0) + verified_resolved = [int](Get-AgentObjectValue $problem "verifiedResolved" 0) + failed = [int](Get-AgentObjectValue $problem "failed" 0) + degraded = [int](Get-AgentObjectValue $problem "degraded" 0) + blocked = [int](Get-AgentObjectValue $problem "blocked" 0) + verifying = [int](Get-AgentObjectValue $problem "verifying" 0) + candidate_ready = [int](Get-AgentObjectValue $problem "candidateReady" 0) + } + occurred_at = (Get-Date -Format o) + } + $pendingPath = Join-Path $pendingDir "$callbackId.json" + $payload | ConvertTo-Json -Depth 10 | Set-Content -Path $pendingPath -Encoding UTF8 + } + + $callbackIdValue = [string](Get-AgentObjectValue $payload "callback_id" "unknown") + $receipt = [ordered]@{ + schemaVersion = "agent99_completion_callback_delivery_v1" + ok = $false + timestamp = (Get-Date -Format o) + callbackId = $callbackIdValue + runId = [string](Get-AgentObjectValue $payload "run_id" "") + traceId = [string](Get-AgentObjectValue $payload "trace_id" "") + endpointConfigured = [bool]$settings.url + tokenPresent = $false + attempts = 0 + accepted = $false + durableReadback = $false + status = "pending" + errorType = $null + httpStatus = $null + responseSchema = $null + awooopRunId = $null + operationReceiptCount = 0 + pendingFile = Split-Path -Leaf $pendingPath + processedFile = $null + secretValueLogged = $false + rawResponseStored = $false + evidenceFile = $null + } + $token = Get-AgentEnvironmentValue $settings.tokenEnv + $receipt.tokenPresent = [bool]$token + if (-not $settings.url) { + $receipt.status = "blocked_endpoint_missing" + } elseif (-not $token) { + $receipt.status = "blocked_token_missing" + } else { + for ($attempt = 1; $attempt -le $settings.maxAttempts; $attempt++) { + $receipt.attempts = $attempt + try { + $response = Invoke-RestMethod -Method Post -Uri $settings.url -Headers @{ + "X-Agent99-Completion-Token" = $token + "X-Project-ID" = [string](Get-AgentObjectValue $payload "project_id" "awoooi") + } -ContentType "application/json; charset=utf-8" -Body ($payload | ConvertTo-Json -Depth 10 -Compress) -TimeoutSec $settings.timeoutSeconds + $receipt.responseSchema = [string](Get-AgentObjectValue $response "schema_version" "") + $receipt.accepted = Convert-AgentBool (Get-AgentObjectValue $response "accepted" $false) + $receipt.durableReadback = Convert-AgentBool (Get-AgentObjectValue $response "durable_readback" $false) + $responseCallbackId = [string](Get-AgentObjectValue $response "callback_id" "") + $responseRunId = [string](Get-AgentObjectValue $response "run_id" "") + $responseTraceId = [string](Get-AgentObjectValue $response "trace_id" "") + $receipt.awooopRunId = [string](Get-AgentObjectValue $response "awooop_run_id" "") + $receipt.operationReceiptCount = [int](Get-AgentObjectValue $response "operation_receipt_count" 0) + $sameIdentity = [bool]( + $responseCallbackId -eq [string](Get-AgentObjectValue $payload "callback_id" "") -and + $responseRunId -eq [string](Get-AgentObjectValue $payload "run_id" "") -and + $responseTraceId -eq [string](Get-AgentObjectValue $payload "trace_id" "") + ) + if ($receipt.accepted -and $receipt.durableReadback -and $sameIdentity -and $receipt.operationReceiptCount -gt 0) { + $processedPath = Join-Path $processedDir (Split-Path -Leaf $pendingPath) + Move-Item -Force $pendingPath $processedPath + $receipt.status = "durable_verified" + $receipt.processedFile = Split-Path -Leaf $processedPath + $receipt.pendingFile = $null + break + } + $receipt.status = "response_not_verified" + } catch { + $receipt.errorType = $_.Exception.GetType().Name + if ($_.Exception.Response -and $_.Exception.Response.StatusCode) { + $receipt.httpStatus = [int]$_.Exception.Response.StatusCode + } + $receipt.status = "delivery_failed" + } + if ($attempt -lt $settings.maxAttempts) { Start-Sleep -Seconds $attempt } + } + } + + if ($receipt.status -ne "durable_verified" -and (Test-Path $pendingPath)) { + (Get-Item $pendingPath).LastWriteTime = Get-Date + } + $receipt.ok = [bool]($receipt.status -eq "durable_verified") + $safeCallbackId = Convert-AgentSafeKey $callbackIdValue + $receiptPath = Join-Path $EvidenceDir "agent99-CompletionCallback-$safeCallbackId-$stamp.json" + $receipt.evidenceFile = Split-Path -Leaf $receiptPath + $receipt | ConvertTo-Json -Depth 10 | Set-Content -Path $receiptPath -Encoding UTF8 + [pscustomobject]$receipt +} + +function Invoke-AgentPendingCompletionCallbacks { + $settings = Get-AgentCompletionCallbackSettings + if (-not $settings.enabled) { + return [pscustomobject]@{ enabled = $false; attemptedCount = 0; verifiedCount = 0; pendingCount = 0; receipts = @() } + } + $pendingDir = Join-Path $Config.agentRoot "state\completion-callbacks\pending" + if (-not (Test-Path $pendingDir)) { + return [pscustomobject]@{ enabled = $true; attemptedCount = 0; verifiedCount = 0; pendingCount = 0; receipts = @() } + } + $eligibleBefore = (Get-Date).AddMinutes(-1 * $settings.retryAfterMinutes) + $files = @(Get-ChildItem -Path $pendingDir -Filter "*.json" -File -ErrorAction SilentlyContinue | + Where-Object { $_.LastWriteTime -le $eligibleBefore } | + Sort-Object LastWriteTime | + Select-Object -First $settings.batchLimit) + $receipts = @($files | ForEach-Object { Invoke-AgentCompletionCallback -PayloadPath $_.FullName }) + [pscustomobject]@{ + enabled = $true + attemptedCount = $receipts.Count + verifiedCount = @($receipts | Where-Object { $_.ok }).Count + pendingCount = @(Get-ChildItem -Path $pendingDir -Filter "*.json" -File -ErrorAction SilentlyContinue).Count + receipts = $receipts + } +} + function Get-AgentDashboardSummary { $status = Get-AgentLatestEvidence "agent99-Status-*.json" $perf = Get-AgentLatestEvidence "agent99-Perf-*.json" @@ -4512,8 +4732,6 @@ function Invoke-AgentQueuedCommands { } $problem = Update-AgentProblemKnowledge $result $result | Add-Member -MemberType NoteProperty -Name problem -Value $problem -Force - $donePath = Join-Path $processedDir ("processed-$id.json") - $result | ConvertTo-Json -Depth 8 | Set-Content -Path $donePath -Encoding UTF8 Remove-Item $runningPath -Force -ErrorAction SilentlyContinue | Out-Null $recordData = $result | Select-Object * if ($replyChatId) { @@ -4524,12 +4742,18 @@ function Invoke-AgentQueuedCommands { } $outcomeSeverity = if ($outcome.state -eq "resolved") { "info" } elseif ($outcome.state -eq "failed") { "critical" } else { "warning" } Record-AgentEvent "operator_command_result" $outcomeSeverity "id=$id mode=$modeName outcome=$($outcome.state) transportExitCode=$exitCode verifierPassed=$($outcome.verifierPassed) sourceEventResolved=$($outcome.sourceEventResolved) evidence=$($latest.path)" $recordData -Alert + $telegramAttempt = @($script:TelegramAttempts | Where-Object { $_.eventType -eq "operator_command_result" } | Select-Object -Last 1) + $completionCallback = Invoke-AgentCompletionCallback -Result $result -TelegramAttempt $(if ($telegramAttempt.Count -gt 0) { $telegramAttempt[0] } else { $null }) + $result | Add-Member -MemberType NoteProperty -Name completionCallback -Value $completionCallback -Force + $donePath = Join-Path $processedDir ("processed-$id.json") + $result | ConvertTo-Json -Depth 10 | Set-Content -Path $donePath -Encoding UTF8 $results += $result } $results } function Invoke-AgentControlTick { + $completionCallbackReplay = Invoke-AgentPendingCompletionCallbacks $processed = Invoke-AgentQueuedCommands $summary = Get-AgentDashboardSummary $dashboard = Write-AgentDashboard $summary @@ -4544,6 +4768,7 @@ function Invoke-AgentControlTick { processedCount = @($processed).Count processed = $processed pendingCount = $pendingCount + completionCallbackReplay = $completionCallbackReplay } Record-AgentEvent "agent_control_tick" "info" "dashboard=$($dashboard.htmlPath) processed=$(@($processed).Count) pending=$pendingCount" $result -NoAlert $result diff --git a/agent99-deploy.ps1 b/agent99-deploy.ps1 index 54a59d737..5fddb65ba 100644 --- a/agent99-deploy.ps1 +++ b/agent99-deploy.ps1 @@ -200,6 +200,14 @@ try { Add-DefaultProperty $config.recoveryCoordinator "maxArtifactAgeSeconds" 600 Add-DefaultProperty $config.recoveryCoordinator "artifactClockSkewSeconds" 90 Add-DefaultProperty $config.recoveryCoordinator "requiredHostAliases" @("99", "110", "111", "112", "120", "121", "188") + Add-DefaultProperty $config "completionCallback" ([pscustomobject]@{}) + Add-DefaultProperty $config.completionCallback "enabled" $true + Add-DefaultProperty $config.completionCallback "url" "https://awoooi.wooo.work/api/v1/agents/agent99/completion-callback" + Add-DefaultProperty $config.completionCallback "tokenEnv" "AGENT99_SRE_RELAY_TOKEN" + Add-DefaultProperty $config.completionCallback "timeoutSeconds" 15 + Add-DefaultProperty $config.completionCallback "maxAttempts" 2 + Add-DefaultProperty $config.completionCallback "batchLimit" 5 + Add-DefaultProperty $config.completionCallback "retryAfterMinutes" 5 Add-DefaultProperty $config "telegram" ([pscustomobject]@{}) Add-DefaultProperty $config.telegram "repeatAfterMinutes" 120 Add-DefaultProperty $config "performance" ([pscustomobject]@{}) diff --git a/agent99.config.99.example.json b/agent99.config.99.example.json index 6bacfc28c..29240f9b0 100644 --- a/agent99.config.99.example.json +++ b/agent99.config.99.example.json @@ -135,6 +135,15 @@ "192.168.0." ] }, + "completionCallback": { + "enabled": true, + "url": "https://awoooi.wooo.work/api/v1/agents/agent99/completion-callback", + "tokenEnv": "AGENT99_SRE_RELAY_TOKEN", + "timeoutSeconds": 15, + "maxAttempts": 2, + "batchLimit": 5, + "retryAfterMinutes": 5 + }, "telegram": { "enabled": true, "botTokenEnv": "AGENT99_TELEGRAM_BOT_TOKEN", diff --git a/agent99.config.example.json b/agent99.config.example.json index 55b2fc10a..505e545f1 100644 --- a/agent99.config.example.json +++ b/agent99.config.example.json @@ -127,6 +127,15 @@ "192.168.0." ] }, + "completionCallback": { + "enabled": true, + "url": "https://awoooi.wooo.work/api/v1/agents/agent99/completion-callback", + "tokenEnv": "AGENT99_SRE_RELAY_TOKEN", + "timeoutSeconds": 15, + "maxAttempts": 2, + "batchLimit": 5, + "retryAfterMinutes": 5 + }, "telegram": { "enabled": true, "botTokenEnv": "AGENT99_TELEGRAM_BOT_TOKEN", diff --git a/apps/api/src/api/v1/agents.py b/apps/api/src/api/v1/agents.py index 5eda603d9..0bebf77a8 100644 --- a/apps/api/src/api/v1/agents.py +++ b/apps/api/src/api/v1/agents.py @@ -26,15 +26,22 @@ Phase 17 技術債修復 (2026-03-26): """ import asyncio +import hmac import json from typing import Any -from fastapi import APIRouter, BackgroundTasks, HTTPException, status +from fastapi import APIRouter, BackgroundTasks, Header, HTTPException, status from fastapi.responses import StreamingResponse from pydantic import BaseModel, Field +from src.core.config import settings +from src.core.context import clear_project_context, set_project_context from src.core.logging import get_logger from src.core.sse import get_publisher +from src.models.agent99_completion import Agent99CompletionCallbackRequest +from src.services.agent99_completion_callback import ( + record_agent99_completion_callback, +) from src.services.agent99_enterprise_ai_automation_work_items import ( load_latest_agent99_enterprise_ai_automation_work_items, ) @@ -1173,6 +1180,55 @@ async def get_agent99_enterprise_ai_automation_work_items() -> dict[str, Any]: ) from exc +@router.post( + "/agent99/completion-callback", + response_model=dict[str, Any], + summary="接收 Agent99 已驗證完成回寫", + description=( + "以現有 Agent99 relay token 驗證 99 主機回寫，將同一 callback/run/trace " + "以冪等方式寫入 AWOOOI operation log 與 AwoooP completed shadow run，" + "並在回應前執行 durable DB readback。payload 禁止 raw log、secret 與完整檔案路徑。" + ), +) +async def post_agent99_completion_callback( + payload: Agent99CompletionCallbackRequest, + x_agent99_completion_token: str | None = Header( + default=None, + alias="X-Agent99-Completion-Token", + ), +) -> dict[str, Any]: + expected_token = settings.AGENT99_SRE_ALERT_RELAY_TOKEN + if not expected_token: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail="agent99_completion_auth_not_configured", + ) + if not x_agent99_completion_token or not hmac.compare_digest( + x_agent99_completion_token, + expected_token, + ): + raise HTTPException( + status_code=status.HTTP_401_UNAUTHORIZED, + detail="agent99_completion_auth_failed", + ) + + context_tokens = set_project_context( + project_id=payload.project_id, + source="agent99_completion_callback", + request_id=payload.callback_id, + ) + try: + receipt = await record_agent99_completion_callback(payload) + finally: + clear_project_context(context_tokens) + if receipt.get("durable_readback") is not True: + raise HTTPException( + status_code=status.HTTP_503_SERVICE_UNAVAILABLE, + detail="agent99_completion_durable_readback_failed", + ) + return receipt + + @router.get( "/awoooi-priority-work-order-readback", response_model=dict[str, Any], diff --git a/apps/api/src/models/agent99_completion.py b/apps/api/src/models/agent99_completion.py new file mode 100644 index 000000000..1ce7e1e08 --- /dev/null +++ b/apps/api/src/models/agent99_completion.py @@ -0,0 +1,73 @@ +"""Bounded Agent99 completion callback contracts.""" + +from __future__ import annotations + +from datetime import datetime +from typing import Literal + +from pydantic import BaseModel, ConfigDict, Field + + +class Agent99TelegramReceipt(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + sent: bool = False + suppressed: bool = False + reason: str | None = Field(default=None, max_length=80) + message_format: str | None = Field(default=None, max_length=80) + visual_sent: bool = False + visual_file: str | None = Field( + default=None, + max_length=180, + pattern=r"^[^/\\]+$", + ) + + +class Agent99ProblemReceipt(BaseModel): + model_config = ConfigDict(extra="forbid") + + key: str | None = Field(default=None, max_length=240) + occurrences: int = Field(default=0, ge=0) + verified_resolved: int = Field(default=0, ge=0) + failed: int = Field(default=0, ge=0) + degraded: int = Field(default=0, ge=0) + blocked: int = Field(default=0, ge=0) + verifying: int = Field(default=0, ge=0) + candidate_ready: int = Field(default=0, ge=0) + + +class Agent99CompletionCallbackRequest(BaseModel): + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + schema_version: Literal["agent99_completion_callback_v1"] + callback_id: str = Field(min_length=3, max_length=180, pattern=r"^[A-Za-z0-9_.:-]+$") + project_id: str = Field(default="awoooi", min_length=1, max_length=64) + run_id: str = Field(min_length=3, max_length=180) + trace_id: str = Field(min_length=3, max_length=180) + work_item_id: str = Field(min_length=3, max_length=180) + alert_id: str | None = Field(default=None, max_length=180) + correlation_key: str | None = Field(default=None, max_length=240) + source: str = Field(default="agent99", max_length=120) + mode: str = Field(min_length=2, max_length=80) + outcome_state: Literal[ + "resolved", + "candidate_ready", + "verifying", + "degraded", + "blocked", + "failed", + ] + controlled_apply: bool = False + transport_ok: bool + verifier_name: str = Field(min_length=2, max_length=160) + verifier_passed: bool + source_event_resolved: bool + source_event_resolution_policy: str = Field(max_length=120) + duration_seconds: float = Field(ge=0, le=3600) + evidence_ref: str = Field(min_length=1, max_length=180, pattern=r"^[^/\\]+$") + alert_kind: str | None = Field(default=None, max_length=120) + alert_service: str | None = Field(default=None, max_length=160) + alert_host: str | None = Field(default=None, max_length=160) + telegram: Agent99TelegramReceipt = Field(default_factory=Agent99TelegramReceipt) + problem: Agent99ProblemReceipt = Field(default_factory=Agent99ProblemReceipt) + occurred_at: datetime diff --git a/apps/api/src/services/agent99_completion_callback.py b/apps/api/src/services/agent99_completion_callback.py new file mode 100644 index 000000000..38073b612 --- /dev/null +++ b/apps/api/src/services/agent99_completion_callback.py @@ -0,0 +1,219 @@ +"""Durable, idempotent Agent99 completion callback writeback.""" + +from __future__ import annotations + +from typing import Any + +from sqlalchemy import text + +from src.db.base import get_db_context +from src.models.agent99_completion import Agent99CompletionCallbackRequest +from src.repositories.alert_operation_log_repository import ( + get_alert_operation_log_repository, +) +from src.services.channel_hub import ( + build_external_alert_provider_event_id, + record_external_alert_event, +) + + +async def _load_callback_readback( + *, + project_id: str, + provider_event_id: str, + callback_id: str, +) -> dict[str, Any] | None: + async with get_db_context(project_id) as db: + result = await db.execute( + text( + """ + SELECT + event_row.event_id::text AS event_id, + event_row.run_id::text AS run_id, + ( + SELECT count(*) + FROM alert_operation_log operation_row + WHERE operation_row.context ->> 'callback_id' = :callback_id + ) AS operation_count, + ( + SELECT count(*) + FROM alert_operation_log operation_row + WHERE operation_row.context ->> 'callback_id' = :callback_id + AND operation_row.event_type = 'EXECUTION_COMPLETED' + ) AS execution_count, + ( + SELECT count(*) + FROM alert_operation_log operation_row + WHERE operation_row.context ->> 'callback_id' = :callback_id + AND operation_row.event_type = 'RESOLVED' + ) AS resolved_count + FROM awooop_conversation_event event_row + WHERE event_row.project_id = :project_id + AND event_row.channel_type = 'internal' + AND event_row.provider_event_id = :provider_event_id + LIMIT 1 + """ + ), + { + "project_id": project_id, + "provider_event_id": provider_event_id, + "callback_id": callback_id, + }, + ) + row = result.mappings().first() + return dict(row) if row else None + + +def _operation_context(payload: Agent99CompletionCallbackRequest) -> dict[str, Any]: + return { + "schema_version": payload.schema_version, + "callback_id": payload.callback_id, + "project_id": payload.project_id, + "run_id": payload.run_id, + "trace_id": payload.trace_id, + "work_item_id": payload.work_item_id, + "alert_id": payload.alert_id, + "correlation_key": payload.correlation_key, + "source": payload.source, + "mode": payload.mode, + "outcome_state": payload.outcome_state, + "controlled_apply": payload.controlled_apply, + "transport_ok": payload.transport_ok, + "verifier_name": payload.verifier_name, + "verifier_passed": payload.verifier_passed, + "source_event_resolved": payload.source_event_resolved, + "source_event_resolution_policy": payload.source_event_resolution_policy, + "duration_seconds": payload.duration_seconds, + "evidence_ref": payload.evidence_ref, + "alert_kind": payload.alert_kind, + "alert_service": payload.alert_service, + "alert_host": payload.alert_host, + "telegram": payload.telegram.model_dump(mode="json"), + "problem": payload.problem.model_dump(mode="json"), + "occurred_at": payload.occurred_at.isoformat(), + "stores_secret": False, + "raw_log_stored": False, + } + + +def _owner_plane(alert_kind: str | None) -> str: + value = (alert_kind or "").lower() + if any(marker in value for marker in ("security", "wazuh", "siem", "cve")): + return "iwooos" + return "awoooi" + + +async def record_agent99_completion_callback( + payload: Agent99CompletionCallbackRequest, +) -> dict[str, Any]: + """Write one callback to the AWOOOI/AwoooP truth chain and read it back.""" + + stage = payload.outcome_state + provider_event_id = build_external_alert_provider_event_id( + "agent99", + payload.callback_id, + stage, + ) + before = await _load_callback_readback( + project_id=payload.project_id, + provider_event_id=provider_event_id, + callback_id=payload.callback_id, + ) + duplicate = before is not None + terminal_resolved = bool( + payload.outcome_state == "resolved" + and payload.verifier_passed + and payload.source_event_resolved + ) + severity = ( + "info" + if payload.outcome_state == "resolved" + else "critical" + if payload.outcome_state == "failed" + else "warning" + ) + context = _operation_context(payload) + event_id = await record_external_alert_event( + project_id=payload.project_id, + provider="agent99", + event_id=payload.callback_id, + stage=stage, + title=f"Agent99 {payload.mode} {payload.outcome_state}", + severity=severity, + namespace="agent99", + target_resource=payload.alert_service or payload.mode, + fingerprint=payload.correlation_key or payload.trace_id, + incident_id=payload.alert_id, + labels={ + "agent": "agent99", + "mode": payload.mode, + "outcome_state": payload.outcome_state, + "verifier_passed": str(payload.verifier_passed).lower(), + "source_event_resolved": str(payload.source_event_resolved).lower(), + "owner_plane": _owner_plane(payload.alert_kind), + }, + annotations={ + "run_id": payload.run_id, + "trace_id": payload.trace_id, + "work_item_id": payload.work_item_id, + "evidence_ref": payload.evidence_ref, + }, + payload=context, + is_duplicate=duplicate, + ) + + repository = get_alert_operation_log_repository() + if int((before or {}).get("execution_count") or 0) == 0: + await repository.append( + "EXECUTION_COMPLETED", + incident_id=payload.alert_id, + actor="agent99_completion_callback", + action_detail=f"{payload.mode}:{payload.outcome_state}", + success=payload.verifier_passed, + context=context, + ) + if terminal_resolved and int((before or {}).get("resolved_count") or 0) == 0: + await repository.append( + "RESOLVED", + incident_id=payload.alert_id, + actor="agent99_completion_callback", + action_detail=f"{payload.mode}:verified_resolved", + success=True, + context=context, + ) + + readback = await _load_callback_readback( + project_id=payload.project_id, + provider_event_id=provider_event_id, + callback_id=payload.callback_id, + ) + operation_count = int((readback or {}).get("operation_count") or 0) + execution_count = int((readback or {}).get("execution_count") or 0) + resolved_count = int((readback or {}).get("resolved_count") or 0) + durable_readback = bool( + readback + and str(readback.get("event_id") or "") == str(event_id) + and readback.get("run_id") + and execution_count > 0 + and (not terminal_resolved or resolved_count > 0) + ) + return { + "schema_version": "agent99_completion_callback_receipt_v1", + "ok": durable_readback, + "status": "duplicate_verified" if duplicate else "recorded_verified", + "accepted": True, + "duplicate": duplicate, + "callback_id": payload.callback_id, + "run_id": payload.run_id, + "trace_id": payload.trace_id, + "work_item_id": payload.work_item_id, + "outcome_state": payload.outcome_state, + "conversation_event_id": str(event_id), + "awooop_run_id": (readback or {}).get("run_id"), + "operation_receipt_count": operation_count, + "execution_receipt_count": execution_count, + "resolved_receipt_count": resolved_count, + "durable_readback": durable_readback, + "stores_secret": False, + "raw_log_stored": False, + } diff --git a/apps/api/tests/test_agent99_completion_callback_api.py b/apps/api/tests/test_agent99_completion_callback_api.py new file mode 100644 index 000000000..cb69f58a6 --- /dev/null +++ b/apps/api/tests/test_agent99_completion_callback_api.py @@ -0,0 +1,270 @@ +from __future__ import annotations + +# ruff: noqa: E402 +import os +from types import SimpleNamespace +from uuid import UUID + +os.environ.setdefault("DATABASE_URL", "postgresql+asyncpg://test:test@localhost/test") + +import pytest +from fastapi import FastAPI +from fastapi.testclient import TestClient +from pydantic import ValidationError + +from src.api.v1 import agents as agents_api +from src.core.config import settings +from src.models.agent99_completion import Agent99CompletionCallbackRequest +from src.services import agent99_completion_callback as callback_service + + +def payload() -> dict: + return { + "schema_version": "agent99_completion_callback_v1", + "callback_id": "agent99-sre-alert-20260711-resolved", + "project_id": "awoooi", + "run_id": "sre-alert-20260711", + "trace_id": "host_recovery:cold-start-gate:no-host", + "work_item_id": "agent99-incident:INC-20260711-001", + "alert_id": "INC-20260711-001", + "correlation_key": "host_recovery:cold-start-gate:no-host", + "source": "agent99-sre-alert-inbox", + "mode": "Recover", + "outcome_state": "resolved", + "controlled_apply": True, + "transport_ok": True, + "verifier_name": "recover_post_condition_v1", + "verifier_passed": True, + "source_event_resolved": True, + "source_event_resolution_policy": "mode_verifier_can_resolve", + "duration_seconds": 9.4, + "evidence_ref": "agent99-Recover-20260711-202342.json", + "alert_kind": "host_recovery", + "alert_service": "cold-start-gate", + "alert_host": "192.168.0.110", + "telegram": { + "sent": True, + "suppressed": False, + "message_format": "incident_card_zh_tw_v4", + "visual_sent": True, + "visual_file": "agent99-card-recovery.png", + }, + "problem": {"occurrences": 2, "verified_resolved": 2}, + "occurred_at": "2026-07-11T20:23:51+08:00", + } + + +def app_client() -> TestClient: + app = FastAPI() + app.include_router(agents_api.router, prefix="/api/v1") + return TestClient(app) + + +def test_completion_callback_fails_closed_without_server_token(monkeypatch) -> None: + monkeypatch.setattr(settings, "AGENT99_SRE_ALERT_RELAY_TOKEN", "") + + response = app_client().post( + "/api/v1/agents/agent99/completion-callback", + json=payload(), + ) + + assert response.status_code == 503 + assert response.json()["detail"] == "agent99_completion_auth_not_configured" + + +def test_completion_payload_rejects_secret_fields_and_full_paths() -> None: + with pytest.raises(ValidationError): + Agent99CompletionCallbackRequest.model_validate( + {**payload(), "bot_token": "must-not-enter-callback"} + ) + with pytest.raises(ValidationError): + Agent99CompletionCallbackRequest.model_validate( + {**payload(), "evidence_ref": r"C:\Wooo\Agent99\evidence\raw.json"} + ) + + +def test_completion_callback_rejects_wrong_token(monkeypatch) -> None: + monkeypatch.setattr(settings, "AGENT99_SRE_ALERT_RELAY_TOKEN", "expected") + + response = app_client().post( + "/api/v1/agents/agent99/completion-callback", + headers={"X-Agent99-Completion-Token": "wrong"}, + json=payload(), + ) + + assert response.status_code == 401 + assert response.json()["detail"] == "agent99_completion_auth_failed" + + +def test_completion_callback_requires_durable_readback(monkeypatch) -> None: + monkeypatch.setattr(settings, "AGENT99_SRE_ALERT_RELAY_TOKEN", "expected") + + async def fake_record(_payload): + return {"ok": False, "durable_readback": False} + + monkeypatch.setattr(agents_api, "record_agent99_completion_callback", fake_record) + response = app_client().post( + "/api/v1/agents/agent99/completion-callback", + headers={"X-Agent99-Completion-Token": "expected"}, + json=payload(), + ) + + assert response.status_code == 503 + assert response.json()["detail"] == "agent99_completion_durable_readback_failed" + + +def test_completion_callback_returns_same_trace_receipt(monkeypatch) -> None: + monkeypatch.setattr(settings, "AGENT99_SRE_ALERT_RELAY_TOKEN", "expected") + + async def fake_record(request): + return { + "schema_version": "agent99_completion_callback_receipt_v1", + "ok": True, + "durable_readback": True, + "callback_id": request.callback_id, + "run_id": request.run_id, + "trace_id": request.trace_id, + "awooop_run_id": "5e5b0080-e8c9-53a7-872b-0fc1ce80f11f", + } + + monkeypatch.setattr(agents_api, "record_agent99_completion_callback", fake_record) + response = app_client().post( + "/api/v1/agents/agent99/completion-callback", + headers={"X-Agent99-Completion-Token": "expected"}, + json=payload(), + ) + + assert response.status_code == 200 + assert response.json()["durable_readback"] is True + assert response.json()["callback_id"] == payload()["callback_id"] + assert response.json()["trace_id"] == payload()["trace_id"] + + +class FakeOperationRepository: + def __init__(self) -> None: + self.events: list[tuple[str, dict]] = [] + + async def append(self, event_type: str, **kwargs): + self.events.append((event_type, kwargs)) + return SimpleNamespace(id=f"operation-{len(self.events)}") + + +@pytest.mark.asyncio +async def test_completion_service_writes_event_run_and_operation_receipts(monkeypatch) -> None: + request = Agent99CompletionCallbackRequest.model_validate(payload()) + event_id = UUID("3c9e8ee7-fb21-5420-afae-ef928f75a58c") + awooop_run_id = "5e5b0080-e8c9-53a7-872b-0fc1ce80f11f" + repository = FakeOperationRepository() + readbacks = iter( + [ + None, + { + "event_id": str(event_id), + "run_id": awooop_run_id, + "operation_count": 2, + "execution_count": 1, + "resolved_count": 1, + }, + ] + ) + event_call: dict = {} + + async def fake_readback(**_kwargs): + return next(readbacks) + + async def fake_record_event(**kwargs): + event_call.update(kwargs) + return event_id + + monkeypatch.setattr(callback_service, "_load_callback_readback", fake_readback) + monkeypatch.setattr(callback_service, "record_external_alert_event", fake_record_event) + monkeypatch.setattr( + callback_service, + "get_alert_operation_log_repository", + lambda: repository, + ) + + receipt = await callback_service.record_agent99_completion_callback(request) + + assert receipt["durable_readback"] is True + assert receipt["awooop_run_id"] == awooop_run_id + assert event_call["provider"] == "agent99" + assert event_call["stage"] == "resolved" + assert [item[0] for item in repository.events] == [ + "EXECUTION_COMPLETED", + "RESOLVED", + ] + assert repository.events[0][1]["context"]["stores_secret"] is False + assert repository.events[0][1]["context"]["raw_log_stored"] is False + + +@pytest.mark.asyncio +async def test_completion_service_is_idempotent(monkeypatch) -> None: + request = Agent99CompletionCallbackRequest.model_validate(payload()) + event_id = UUID("3c9e8ee7-fb21-5420-afae-ef928f75a58c") + existing = { + "event_id": str(event_id), + "run_id": "5e5b0080-e8c9-53a7-872b-0fc1ce80f11f", + "operation_count": 2, + "execution_count": 1, + "resolved_count": 1, + } + repository = FakeOperationRepository() + event_call: dict = {} + + async def fake_readback(**_kwargs): + return existing + + async def fake_record_event(**kwargs): + event_call.update(kwargs) + return event_id + + monkeypatch.setattr(callback_service, "_load_callback_readback", fake_readback) + monkeypatch.setattr(callback_service, "record_external_alert_event", fake_record_event) + monkeypatch.setattr( + callback_service, + "get_alert_operation_log_repository", + lambda: repository, + ) + + receipt = await callback_service.record_agent99_completion_callback(request) + + assert receipt["status"] == "duplicate_verified" + assert receipt["duplicate"] is True + assert event_call["is_duplicate"] is True + assert repository.events == [] + + +@pytest.mark.asyncio +async def test_completion_service_repairs_partial_terminal_receipt(monkeypatch) -> None: + request = Agent99CompletionCallbackRequest.model_validate(payload()) + event_id = UUID("3c9e8ee7-fb21-5420-afae-ef928f75a58c") + before = { + "event_id": str(event_id), + "run_id": "5e5b0080-e8c9-53a7-872b-0fc1ce80f11f", + "operation_count": 1, + "execution_count": 1, + "resolved_count": 0, + } + after = {**before, "operation_count": 2, "resolved_count": 1} + readbacks = iter([before, after]) + repository = FakeOperationRepository() + + async def fake_readback(**_kwargs): + return next(readbacks) + + async def fake_record_event(**_kwargs): + return event_id + + monkeypatch.setattr(callback_service, "_load_callback_readback", fake_readback) + monkeypatch.setattr(callback_service, "record_external_alert_event", fake_record_event) + monkeypatch.setattr( + callback_service, + "get_alert_operation_log_repository", + lambda: repository, + ) + + receipt = await callback_service.record_agent99_completion_callback(request) + + assert receipt["durable_readback"] is True + assert [item[0] for item in repository.events] == ["RESOLVED"] diff --git a/apps/api/tests/test_agent99_enterprise_ai_automation_work_items_api.py b/apps/api/tests/test_agent99_enterprise_ai_automation_work_items_api.py index dc1f64a4d..9bf2dee9f 100644 --- a/apps/api/tests/test_agent99_enterprise_ai_automation_work_items_api.py +++ b/apps/api/tests/test_agent99_enterprise_ai_automation_work_items_api.py @@ -31,7 +31,7 @@ def test_agent99_enterprise_work_items_loader_returns_complete_scope() -> None: "agent99_enterprise_ai_automation_work_items_v1" ) assert payload["scope_complete"] is False - assert payload["current_p0"]["id"] == "AG99-P0-001" + assert payload["current_p0"]["id"] == "AG99-P0-006" assert payload["summary"] == { "host_count": 7, "product_count": 12, @@ -40,18 +40,18 @@ def test_agent99_enterprise_work_items_loader_returns_complete_scope() -> None: "p0_count": 14, "p1_count": 8, "p2_count": 3, - "in_progress_count": 10, + "in_progress_count": 9, "in_progress_blocked_count": 0, "planned_count": 14, - "complete_count": 1, + "complete_count": 2, "current_p0_count": 1, } assert payload["next_execution_order"][:5] == [ - "AG99-P0-001", "AG99-P0-006", "AG99-P0-003", "AG99-P0-004", "AG99-P0-005", + "AG99-P0-007", ] assert payload["next_execution_order"][-1] == "AG99-P0-011" @@ -60,7 +60,7 @@ def test_agent99_enterprise_projection_is_bounded() -> None: payload = load_latest_agent99_enterprise_ai_automation_work_items() projection = build_agent99_enterprise_priority_projection(payload) - assert projection["current_p0"]["id"] == "AG99-P0-001" + assert projection["current_p0"]["id"] == "AG99-P0-006" assert projection["coverage"] == { "host_count": 7, "product_count": 12, @@ -79,7 +79,7 @@ def test_priority_readback_projects_agent99_enterprise_order() -> None: payload = load_latest_awoooi_priority_work_order_readback() projection = payload["agent99_enterprise_ai_automation"] - assert projection["current_p0"]["id"] == "AG99-P0-001" + assert projection["current_p0"]["id"] == "AG99-P0-006" assert projection["work_item_counts"]["p0"] == 14 assert ( payload["source_refs"]["agent99_enterprise_ai_automation_work_items_api"] @@ -124,7 +124,7 @@ def test_agent99_enterprise_work_items_endpoint_returns_snapshot() -> None: assert response.status_code == 200 payload = response.json() - assert payload["current_p0"]["id"] == "AG99-P0-001" + assert payload["current_p0"]["id"] == "AG99-P0-006" assert payload["summary"]["host_count"] == 7 assert payload["summary"]["product_count"] == 12 assert payload["summary"]["public_surface_count"] == 23 diff --git a/docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json b/docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json index b7e123894..dddbc06b4 100644 --- a/docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json +++ b/docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json @@ -1,7 +1,7 @@ { "schema_version": "agent99_enterprise_ai_automation_work_items_v1", - "generated_at": "2026-07-11T20:14:33+08:00", - "status": "in_progress_p0_agent99_false_green_and_lifecycle_closure", + "generated_at": "2026-07-11T20:34:00+08:00", + "status": "in_progress_p0_agent99_canonical_completion_callback", "scope_complete": false, "north_star": "Make Windows host 192.168.0.99 the policy-controlled on-prem AI execution node for all hosts, VMs, services, products, sites, tools, packages, alerts and recovery workflows, with external monitoring when host 99 is unavailable.", "canonical_sources": { @@ -17,22 +17,22 @@ "runtime_reboot_slo_readback": "https://awoooi.wooo.work/api/v1/agents/reboot-auto-recovery-slo-scorecard" }, "current_p0": { - "id": "AG99-P0-001", - "title": "Agent99 runtime truth, false-green prevention and visible lifecycle receipt", - "reason": "The 2026-07-11 live audit proved that Agent99 runs real recovery work, but the operator cannot reliably see the closure: required inbox evidence can fail safe-summary parsing while preflight remains green, a successful Recover produces zero Telegram attempts because info lifecycle events are filtered out, and dedupe suppression leaves no explicit delivery receipt. Runtime truth and one-run lifecycle visibility must be corrected before adding more automation breadth.", - "implementation_status": "regression_reopened_source_fix_in_progress_runtime_deploy_and_recover_verifier_pending", - "next_action": "Deploy the parse-aware preflight and recovery lifecycle card to Windows 99, run one production-principal Recover, and require one explicit Telegram sent-or-dedupe receipt plus a parse-clean live preflight." + "id": "AG99-P0-006", + "title": "Agent99 canonical completion callback and cross-system incident closure", + "reason": "P0-001 is now runtime-verified on Windows 99: required evidence is fresh, parseable and content-healthy; a production-principal Recover passed all seven phases and produced a real Telegram visual receipt. The next missing link is canonical Agent99-to-AWOOOI/AwoooP/IWOOOS writeback, because Telegram delivery alone is not a durable incident closure.", + "implementation_status": "api_and_agent99_callback_source_ready_39_tests_passed_production_cd_and_runtime_callback_pending", + "next_action": "Commit the authenticated idempotent callback source, wait for the coordinated Gitea push slot, deploy through Gitea CD, then replay one SRE alert and require identical callback/run/trace plus AWOOOI operation and AwoooP run readback." }, "current_runtime_truth": { - "observed_at": "2026-07-11T17:37:00+08:00", + "observed_at": "2026-07-11T20:25:26+08:00", "awoooi_health_http_status": 200, "awoooi_health_state": "recovered_late_currently_healthy", "awoooi_runtime_build_commit_sha": "4a4b95e5135e8cf71d807a54700a962f5490088f", "gitea_main_observed_sha": "f102616f0", "gitea_main_deployed_source_sha": "4a4b95e5135e8cf71d807a54700a962f5490088f", "runtime_work_items_generated_at": "2026-07-10T19:40:00+08:00", - "windows99_agent99_manifest_observed_at": "2026-07-11T17:31:00+08:00", - "windows99_agent99_source_revision": "442b319bb80e009291ced7afa4f704cbe3b64886", + "windows99_agent99_manifest_observed_at": "2026-07-11T20:20:51+08:00", + "windows99_agent99_source_revision": "06509e071c8b7c538feb112e4e4774ee22abf7d1", "windows99_agent99_runtime_matched": true, "windows99_agent99_runtime_file_count": 14, "windows99_agent99_runtime_mismatch_count": 0, @@ -41,11 +41,23 @@ "windows99_agent99_relay_listener_count": 1, "windows99_agent99_alert_incoming_count": 0, "windows99_agent99_command_pending_count": 0, + "windows99_agent99_live_preflight_green": true, + "windows99_agent99_required_evidence_fresh_count": 7, + "windows99_agent99_required_evidence_usable_count": 7, + "windows99_agent99_required_evidence_parse_failure_count": 0, + "windows99_agent99_required_evidence_content_failure_count": 0, + "windows99_agent99_recovery_run_id": "agent99-recovery-20260711-202159-3404b5b3", + "windows99_agent99_recovery_completed": true, + "windows99_agent99_recovery_elapsed_seconds": 42.9, + "windows99_agent99_recovery_phase_count": 7, + "windows99_agent99_recovery_telegram_sent": true, + "windows99_agent99_recovery_visual_sent": true, + "agent99_completion_callback_runtime_status": "source_ready_production_api_and_windows_client_not_yet_deployed", "windows99_agent99_startup_recovery_execution_limit": "PT20M", "windows99_vmware_running_guests": "5/5", "public_surface_matrix": "23/23_http_2xx_or_expected_3xx", "reboot_slo_status": "slo_breached_recovered_late", - "reboot_slo_readiness_percent": 87, + "reboot_slo_readiness_percent": 89, "reboot_slo_primary_blocker": "all_required_hosts_not_in_10_minute_reboot_window", "reboot_slo_active_blockers": [ "all_required_hosts_not_in_10_minute_reboot_window", @@ -272,7 +284,7 @@ "id": "AG99-P0-001", "order": 1, "priority": "P0", - "status": "in_progress", + "status": "complete", "title": "Agent outcome truth, source-runtime reconciliation and false-green prevention", "owner_lane": "agent99_control_plane", "dependencies": [], @@ -289,9 +301,9 @@ "callback": "Write final outcome to AWOOOI incident and AwoooP Run/Verifier using the same trace id.", "acceptance": "No false-resolved regression cases; source/runtime drift count is zero; unresolved evidence stays degraded or blocked.", "completion_receipt": { - "source_revision": "5ea4405c2a75b51244da168a4834d27e24d74ee6", + "source_revision": "06509e071c8b7c538feb112e4e4774ee22abf7d1", "runtime_manifest_matched": true, - "runtime_manifest_file_count": 7, + "runtime_manifest_file_count": 14, "runtime_manifest_mismatch_count": 0, "self_health_runtime_manifest_severity": "ok", "outcome_self_test_ok": true, @@ -300,19 +312,31 @@ "alert_replay_state": "verifying", "alert_replay_verified_resolved": 0, "alert_replay_source_event_resolved": false, - "telegram_sent_count": 0, - "rollback": "C:\\Wooo\\Agent99\\rollback\\p0-001-pre-5ea4405c", + "telegram_sent_count": 1, + "telegram_visual_sent_count": 1, + "required_evidence_usable_count": 7, + "required_evidence_parse_failure_count": 0, + "required_evidence_content_failure_count": 0, + "recovery_run_id": "agent99-recovery-20260711-202159-3404b5b3", + "recovery_completed": true, + "recovery_elapsed_seconds": 42.9, + "recovery_phase_count": 7, + "rollback": "C:\\Wooo\\Agent99\\deploy\\backup-20260711-202021", "evidence": [ "C:\\Wooo\\Agent99\\evidence\\agent99-OutcomeContractSelfTest-20260710-161302.json", "C:\\Wooo\\Agent99\\evidence\\agent99-SelfCheck-20260710-161350.json", "C:\\Wooo\\Agent99\\queue\\processed\\processed-p0-001-operator-verifier-20260710-1615.json", "C:\\Wooo\\Agent99\\queue\\processed\\processed-p0-001-alert-verifier-20260710-1615.json", - "C:\\Wooo\\Agent99\\evidence\\agent99-ControlTick-20260710-161919.json" + "C:\\Wooo\\Agent99\\evidence\\agent99-ControlTick-20260710-161919.json", + "C:\\Wooo\\Agent99\\evidence\\agent99-deploy-20260711-202021.json", + "C:\\Wooo\\Agent99\\evidence\\agent99-Recover-20260711-202159.json", + "C:\\Wooo\\Agent99\\evidence\\telegram-cards\\agent99-card-recovery_slo_result-20260711-202159.png" ] }, "regression_reopened_at": "2026-07-11T20:02:00+08:00", "regression_evidence": ["Windows 99 live preflight returned green while latest SreAlertInbox and TelegramInbox safe summaries reported RuntimeException", "latest production-principal Recover was verifier-green but telegramAttemptCount=0", "dedupe suppression returned without a Telegram attempt receipt"], - "next_action": "Deploy the parse-aware evidence gate, explicit lifecycle info alert and dedupe receipt; rerun production-principal Recover and keep this item open until runtime readback is parse-clean and notification-visible." + "closed_at": "2026-07-11T20:25:26+08:00", + "next_action": "Keep parse/content health and lifecycle notification receipts under regression monitoring; continue with AG99-P0-006." }, { "id": "AG99-P0-002", @@ -432,8 +456,8 @@ "verifier": "Read back identical trace, final state and verifier evidence from AWOOOI and AwoooP.", "callback": "This work item creates the canonical callback contract.", "acceptance": "One trace is visible from alert receipt through final verifier, notification and recurrence counters.", - "implementation_progress": "The first closure slice is being implemented in Agent99 runtime: Recover now has a dedicated Traditional-Chinese lifecycle summary, a phase timeline image, a successful info-event delivery policy and an explicit dedupe receipt. Canonical authenticated AWOOOI/AwoooP/IWOOOS writeback remains open.", - "next_action": "Verify the Windows 99 recovery lifecycle receipt, then define the canonical callback schema and idempotent authenticated AWOOOI/AwoooP/IWOOOS endpoint." + "implementation_progress": "Recover lifecycle visibility is runtime verified on Windows 99. Source now adds an authenticated callback endpoint using the existing Agent99 relay token, idempotent AWOOOI operation receipts, an AwoooP completed shadow run, durable DB readback, bounded Agent99 pending retries and no-secret payload validation. Focused callback/recovery tests pass 39/39; production Gitea CD and one live same-trace replay remain open.", + "next_action": "Commit the callback source, obtain the coordinated single-writer push slot, deploy API and Agent99 client in order, then replay one SRE alert and verify AWOOOI/AwoooP/IWOOOS plus Telegram receipts share the same trace." }, { "id": "AG99-P0-007", @@ -791,14 +815,13 @@ "p0_count": 14, "p1_count": 8, "p2_count": 3, - "in_progress_count": 10, + "in_progress_count": 9, "in_progress_blocked_count": 0, "planned_count": 14, - "complete_count": 1, + "complete_count": 2, "current_p0_count": 1 }, "next_execution_order": [ - "AG99-P0-001", "AG99-P0-006", "AG99-P0-003", "AG99-P0-004", diff --git a/docs/runbooks/REBOOT-RECOVERY-SOP.md b/docs/runbooks/REBOOT-RECOVERY-SOP.md index 51464e779..6742acde1 100644 --- a/docs/runbooks/REBOOT-RECOVERY-SOP.md +++ b/docs/runbooks/REBOOT-RECOVERY-SOP.md @@ -79,6 +79,8 @@ Windows 99 VMware task 已經由 `scripts/reboot-recovery/windows99-vmware-autos 2026-07-11 runtime closure：installer runs `host112-guest-recovery-20260711T193137+0800-35698` 與 `host112-guest-recovery-20260711T195449+0800-43505` 已完成備份、受限公鑰、exact-command sudo、timer 與 durable rollback installer，未 reboot、未改 VM power、未讀 secret。110 artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260711-200338` 讀回 112 `systemd=running`、Console/Wazuh/timer/guest `1/1/1/1`、五項 Prometheus gauges `5/5`、`host112_* blocker=0`。兩條 112 告警規則 health `2/2 ok` 且 healthy state 為 inactive；source/canonical/active/runtime rule SHA 均為 `86fd0e56cb7633da2a5a09960b8bf2dd61ee917c55072b5a16ba68577e8379cc`。Windows99 verifier 同輪讀到 `VM_PROCESS alias=112 process_id=22288 session_id=0 owner=NT_AUTHORITY\SYSTEM ui_truth=runtime_running_gui_session_isolated`，因此 operator power truth 固定信任 VMX process + guest readiness，不信任跨 principal 的互動式 VMware UI。Agent99 `Recover` 也已把 112 納入必要 phase 並完成 production-principal verifier；完整 receipt 見 `docs/operations/host112-guest-recovery-20260711.snapshot.json`。Agent99 tests `49 passed`、reboot-recovery tests `142 passed`。目前 scorecard 僅保留 `all_required_hosts_not_in_10_minute_reboot_window`、`fresh_all_host_reboot_event_missing`、`host_boot_observation_older_than_target_window`；它們只能由下一次真實全主機 600 秒 drill 關閉。 +2026-07-11 20:25 Agent99 false-green closure：Windows runtime 已原子部署 `06509e071c8b7c538feb112e4e4774ee22abf7d1`，deploy receipt `agent99-deploy-20260711-202021.json`、rollback `backup-20260711-202021`、manifest `14/14` matched。新版 preflight 為 tasks `9/9`、required evidence fresh/usable `7/7`、parse/content failure `0`。production-principal run `agent99-recovery-20260711-202159-3404b5b3` 在 42.9 秒完成 5/5 VM、5/5 host、5/5 public 與七個 phase，TG `sent=1`、`visualSent=1`，實際圖卡 `agent99-card-recovery_slo_result-20260711-202159.png` 為 1200×760 phase timeline；下一輪同狀態以 `dedupe_window` 抑制且留下 receipt。這關閉 Agent99 可見性假綠，不改寫原始全主機 T+10 breach。 + ### 2026-07-10 P0 Agent99 Transport / Verified Remediation 覆蓋 99 的 Agent99 是全主機恢復控制節點。Windows 開機後由既有 Agent99 scheduled tasks 啟動；`Status` 讀到 boot identity 變更時，只能用 single-flight queue 觸發一次 `Recover`，並以 10 分鐘 cooldown 避免併發恢復。每次執行必須寫入 `bootState`、`recoveryTrigger`、`recoveryCoordinator` 與 `recoverySlo` evidence。只有實際全主機重啟事件、七主機 reachability、必要服務、public routes、資料 freshness、備份、edge fallback、VMware/Windows policy 與 188 hygiene 全部在同一個 10 分鐘 window 通過，才可宣稱 reboot recovery SLO 達標。 diff --git a/docs/workplans/2026-07-10-agent99-enterprise-ai-automation-master-plan.md b/docs/workplans/2026-07-10-agent99-enterprise-ai-automation-master-plan.md index c2f3a69d9..133ee82bf 100644 --- a/docs/workplans/2026-07-10-agent99-enterprise-ai-automation-master-plan.md +++ b/docs/workplans/2026-07-10-agent99-enterprise-ai-automation-master-plan.md @@ -1,6 +1,6 @@ # Agent99 全域 AI 自動化主計畫 -> 建立時間：2026-07-10 15:05 CST；最近更新：2026-07-11 20:14 CST +> 建立時間：2026-07-10 15:05 CST；最近更新：2026-07-11 20:34 CST > 狀態：P0 執行中，尚未達成全域 AI 自動化 > 機器可讀總帳：`docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json` > 歷史需求來源：`docs/workplans/2026-07-02-commander-inserted-requirements-priority-ledger.md` @@ -97,13 +97,13 @@ ## 5. 更新後的唯一執行順序 -2026-07-11 20:14 live audit 重新校正後，機器可讀執行順序固定為：`AG99-P0-001 → P0-006 → P0-003 → P0-004 → P0-005 → P0-007 → P0-008 → P0-009 → P0-013 → P0-012 → P0-014 → P0-010 → P0-011`。先關閉 Agent99 假綠與看不見的執行結果，再擴充 ingress、evidence/candidate、verifier 與各 domain 自動化。下表的數字是穩定 work-item ID，不再被誤當目前執行順位；`next_execution_order` 才是 runtime projection 的唯一順位來源，且必須包含每個未完成 P0 恰好一次。 +2026-07-11 20:34 runtime 驗證後，機器可讀執行順序固定為：`AG99-P0-006 → P0-003 → P0-004 → P0-005 → P0-007 → P0-008 → P0-009 → P0-013 → P0-012 → P0-014 → P0-010 → P0-011`。`P0-001` 的假綠與看不見結果已由 99 runtime receipt 關閉，現在先完成 canonical callback，再擴充 ingress、evidence/candidate、verifier 與各 domain 自動化。下表的數字是穩定 work-item ID，不再被誤當目前執行順位；`next_execution_order` 才是 runtime projection 的唯一順位來源，且必須包含每個未完成 P0 恰好一次。 ### Wave 0：先修正 AI 決策真相 | 順序 | ID | 工作 | 狀態 | 驗收 | | --- | --- | --- | --- | --- | -| 1 | AG99-P0-001 | 修正 `exitCode=0 => resolved`、evidence parse 假綠，對齊 99 runtime script 與 source SHA | **重新開啟；current P0** | Production 99 outcome self-test 通過；required evidence 必須 fresh 且可解析；Recover 必須有 sent 或明確 suppressed receipt；runtime manifest matched、drift `0` | +| 1 | AG99-P0-001 | 修正 `exitCode=0 => resolved`、evidence parse 假綠，對齊 99 runtime script 與 source SHA | **完成；99 runtime `06509e071`** | required evidence `7/7` fresh+parsed+content healthy；Recover 7/7 phase verified；TG `sent=1`、`visualSent=1`；runtime manifest 14/14 matched | | 2 | AG99-P0-002 | 修正 alert taxonomy、wrong-domain routing、bridge-before-grouping、actionable Telegram card 漏接與重複事件 | **完成；99 runtime `34e32f6e`、14/14 matched** | Alertmanager、AI card、ACTION REQUIRED、CI/CD failure 共用 dispatcher；`suggestedMode`/kind 優先；5 分鐘先 group；同 fingerprint 只建立一個 active incident；保留 acceptance receipt | | 3 | AG99-P0-003 | 強化 ingress：relay token 必填、Telegram chat+sender allowlist、idempotency/single consumer | 待辦 | 未認證 LAN request 拒絕；producer `auto_repair`/policy 被保留；所有 request 可追 trace | | 4 | AG99-P0-004 | 新增 EvidenceCollect / RepairCandidate / MCP lane | 待辦 | 缺 evidence 的事件會建立可查 candidate，不再錯送 Perf/AwoooRepair；`INC-*` 可從收件追到 candidate | @@ -113,7 +113,7 @@ | 順序 | ID | 工作 | 狀態 | 驗收 | | --- | --- | --- | --- | --- | | 5 | AG99-P0-005 | 模式專屬 verifier、真 rollback、cooldown/circuit-breaker | 待辦 | Recover/Perf/LoadShed/Harbor/AWOOOI/backup/provider 各有 post-condition；失敗不寫成功 cooldown | -| 6 | AG99-P0-006 | Agent99 → AWOOOI/AwoooP/IWOOOS completion callback | **進行中；先關閉 Recover 可見生命週期 receipt** | incident/Run/Work Item/Verifier/recurrence 同一 trace 收斂；Telegram 可讀回報只從結果物件產生 | +| 6 | AG99-P0-006 | Agent99 → AWOOOI/AwoooP/IWOOOS completion callback | **進行中；current P0** | incident/Run/Work Item/Verifier/recurrence 同一 trace 收斂；Telegram 可讀回報只從結果物件產生 | | 7 | AG99-P0-007 | 全主機 10 分鐘 reboot state machine | **進行中；本輪 breach 後已 late recovery，L1/L2 runtime 已整合，L0/600 秒重測待完成** | L0 off-LAN detection/OOB power、L1 Agent99/VMware、L2 110 full-SOP 必須以同一 incident 閉環；首可用/全綠時間與 blocker/ETA 可查 | | 8 | AG99-P0-008 | 99 VMware/VMX/autostart/lock 與 Windows Update no-auto-reboot | **本次 boot runtime green；controlled apply/rollback/verifier 完成，待下次 reboot 驗證 startup 時序** | 110/188/112/120/121 VMX、autostart task、guest running 與 no-auto-reboot verifier green；111 由實體 Mac readback 驗證 | @@ -200,6 +200,13 @@ - 同一 preflight 對最新 SRE/Telegram inbox safe summary 顯示 `RuntimeException`，卻仍因檔案時間新而回傳綠燈。這是明確 false-green regression，不可用更多 dashboard 掩蓋。 - current P0 因此重新定位為 `AG99-P0-001`，第二順位是 `AG99-P0-006`：先把 required evidence 改成 fresh + parseable、Recover 成功事件形成繁中生命週期卡與 phase timeline、dedupe 也留下 receipt；驗證後才回到 secure ingress、candidate、verifier 與 reboot/各 domain 主線。 +### 2026-07-11 20:34 P0-001 runtime closure 與 P0-006 source + +- Agent99 原子部署 receipt `agent99-deploy-20260711-202021.json` 回報 source revision `06509e071c8b7c538feb112e4e4774ee22abf7d1`、14/14 runtime files matched，rollback 是 `backup-20260711-202021`。 +- 新版 live preflight 在 20:20:51 回報 9/9 tasks、relay=1、queue=0，required evidence `fresh=7/7`、`usable=7/7`、parse/content failure 都是 0；不再用檔案時間掩蓋 inbox JSON 解析失敗。 +- Production-principal Recover `agent99-recovery-20260711-202159-3404b5b3` 完成 5/5 VM、5/5 host、5/5 public 與 7/7 recovery phases；42.9 秒、coordinator verified、service-recovery only。TG receipt 是 `sent=1`、`visualSent=1`，實際 1200×760 圖卡顯示七階段時間軸。後續同狀態 run 以 `dedupe_window` suppressed 並保留 receipt。 +- `P0-006` source 已新增 token-authenticated callback、idempotent channel event、AWOOOI operation receipts、AwoooP completed shadow run、durable DB readback，以及 99 pending/processed retry queue；payload 禁止 secret、raw log 與完整 evidence path。Focused tests 39 passed、ruff green。此段尚未經 Gitea CD 與 production replay，所以維持 in progress。 + ## 7. 工作項強制欄位與治理 每個工作項必須先有：`id`、priority/order/status、scope、owner lane、dependencies、acceptance 與 next action。任何工作項升到 `candidate_ready`、`in_progress` 或 `executing` 前，還必須補齊 problem、trigger、evidence sources、AI role、action mode、controlled actions、prohibited actions、rollback、mode-specific verifier、callback、evidence refs 與 last update；缺一不可執行。 @@ -215,4 +222,4 @@ ## 8. 立即下一步 -目前唯一 P0 是 `AG99-P0-001`。立即動作是把 parse-aware preflight、Recover 生命週期卡、phase timeline 與 dedupe receipt 原子部署到 99，再由 production principal 跑一次 Recover；只有 runtime manifest matched、preflight parse-clean、Telegram sent 或 dedupe-suppressed receipt 明確存在才前進。後續固定依序：`P0-006 → P0-003 → P0-004 → P0-005 → P0-007 → P0-008 → P0-009 → P0-013 → P0-012 → P0-014 → P0-010 → P0-011`。main push 只在唯一 writer slot 進行；沒有 slot 時保留 local commit，不讓推送協調打斷 runtime 主線。 +目前唯一 P0 是 `AG99-P0-006`。立即動作是提交 callback source，依單一 writer 協調取得 main push slot，讓 API 先經 Gitea CD 上線；再部署 99 callback client，重播一個受控 SRE alert，要求同一 callback/run/trace 在 AWOOOI operation log、AwoooP Run、IWOOOS owner plane、Agent99 local receipt 與 Telegram 都可讀回。後續固定依序：`P0-003 → P0-004 → P0-005 → P0-007 → P0-008 → P0-009 → P0-013 → P0-012 → P0-014 → P0-010 → P0-011`。main push 只在唯一 writer slot 進行；沒有 slot 時保留 local commit，不讓推送協調打斷 runtime 主線。