ops(agent99): preserve callback recovery closure
This commit is contained in:
ogt
@@ -772,3 +772,11 @@ Agent99 now uses a source-trusted deployment path, a dedicated bounded SSH trans
|
||||
- Canonical completion is not Telegram delivery. Agent99 posts a bounded `agent99_completion_callback_v1` payload authenticated by the existing `AGENT99_SRE_RELAY_TOKEN`; raw logs, secrets and full evidence paths are rejected.
|
||||
- The API writes an idempotent Agent99 channel event, AWOOOI `EXECUTION_COMPLETED`/verified `RESOLVED` operation receipts and an AwoooP completed shadow run. Agent99 moves a callback from `state\completion-callbacks\pending` to `processed` only after the API returns the same callback/run/trace and `durable_readback=true` with operation receipts.
|
||||
- Callback failures retry in bounded batches after cooldown and never repeat the repair action. The callback source is not production-complete until Gitea CD, API runtime readback, Windows client deploy and one same-trace SRE replay all pass.
|
||||
|
||||
## 2026-07-14 Callback Identity And Recurrence Closure
|
||||
|
||||
- `alert_operation_log.incident_id` is limited to 30 characters. Agent99 must prefer `result.identity.incident_id`; a longer fallback is converted to deterministic `INC-AG99-<21 hex>`. The API independently applies the same bound before writing operation receipts. A long external alert id remains in bounded context only and must never be used as the operation-log incident key.
|
||||
- Existing pending payloads are repaired only with `scripts/reboot-recovery/agent99-completion-callback-incident-id-migration.ps1`. Check mode must map every `run_id` to one processed result and require a canonical `INC-*` id with no blocker. Apply mode uses immutable backups, atomic replacement, post-readback and all-or-nothing rollback; receipts record the prior length and SHA-256, not the raw external id.
|
||||
- Replay uses the existing `Wooo-Agent99-Control-Loop`, batch limit, authentication and durable verifier. A file moves from pending to processed only when the API returns the same callback/run/trace, `accepted=true`, `durable_readback=true` and `operation_receipt_count>0`. Replay must not repeat the repair action or emit a second Telegram lifecycle card.
|
||||
- Production recurrence is closed only when the latest source stage is `resolved`, the linked run is `completed`, and the same callback has successful `EXECUTION_COMPLETED` and `RESOLVED` receipts with `verifier_passed=true` and `source_event_resolved=true`. A reopened event invalidates the prior callback for current-state closure and queues automated re-verification; it does not default to a human gate.
|
||||
- Live receipt: Windows runtime source `568a72e7218fee8a588c31fe834d6751217640b0` deployed with 14/14 files and 9/9 tasks. Migration receipt `agent99-callback-incident-id-migration-4eed48cd40db7ee1636b.json` verified 29/29 payloads. Replay receipt `agent99-callback-replay-ba20da2c812f7ba41230.json` reduced pending callbacks from 13 to 0 in batches 5/5/3. Final readback was queue 0, callback pending 0, callback processed 40; the latest real cold-start events were resolved with durable callback readback and two operation receipts each.
|
||||
|
||||
@@ -65,6 +65,10 @@ L0 source contract 是 `scripts/reboot-recovery/external-l0-recovery-supervisor.
|
||||
|
||||
部署前 gate 是：PowerShell parse、Agent99 contract、synthetic fresh/blocked fixture、runtime staging hash；部署後 gate 是：`scripts/reboot-recovery/agent99-live-preflight.ps1` 回 `9/9` tasks、relay listener、queue zero、required evidence fresh、runtime manifest matched，再由 production S4U principal 觸發 `Recover`。Administrator 直接跑 launcher 不能取代 production-principal verifier。
|
||||
|
||||
2026-07-14 callback closure：Agent99 runtime `568a72e7218fee8a588c31fe834d6751217640b0` 已經原子部署，remote deploy receipt 為 `agent99-remote-deploy-fc85e242f7e8f82324c4.json`，manifest 14/14、task 9/9、relay generation 更新且 live preflight green。正式 Startup-Recovery run `agent99-recovery-20260714-165036-d0e26090` 驗證 5/5 VM、6/6 host、5/5 public 與八個 phase，12.5 秒、coordinator verified、Telegram 人類可讀圖卡已送達；這是 service recovery，不冒充 fresh full-host reboot window。
|
||||
|
||||
同輪發現 AWOOOI callback 503 不是 transport 或 token 問題，而是 51 字元外部 alert id 寫入 `alert_operation_log.incident_id varchar(30)`。sender 已改用 canonical `identity.incident_id`，API receiver 另有 deterministic 30 字元防禦。29 筆舊 pending payload 經 check、immutable backup、atomic migration、post-verifier 後，以既有 ControlTick 分批重送；最終 pending queue/callback 都是 0、processed callback 40。最新真實 cold-start incident 在 Agent99 端為 `resolved + verifierPassed + sourceEventResolved`，callback 為 `durable_verified + accepted + operationReceiptCount=2`，AWOOOI public incident readback也是 resolved。AwoooP recurrence 若仍顯示 `run_completed_no_repair`，固定判定為 read-model false-red；必須部署 callback-operation reconciliation，不能重跑修復或改成人工處理。
|
||||
|
||||
2026-07-11 20:09 final runtime receipt：Agent99 已原子 promote 為 `d0e364bfa42947190cf1e88d1d85623e23f84858`，deploy evidence `C:\Wooo\Agent99\evidence\agent99-deploy-20260711-200939.json`、rollback backup `C:\Wooo\Agent99\deploy\backup-20260711-200939`；runtime manifest `14/14`、mismatch `0`。production-principal evidence `C:\Wooo\Agent99\evidence\agent99-Recover-20260711-200320.json` 顯示七個 phases 全部 `ok`，新增必要 phase `host112_guest_readiness`，before/after verifier 均為 ready、`applyAttempted=false`，transport 為 direct serialized，recovery completed、coordinator verified，elapsed `44.6s`，且正確維持 `scope=service_recovery`、`rebootSloClaimed=false`。SRE routing self-test `17/17`、Telegram ingest self-test `16/16`，後者不寫 alert、不送 Telegram。這是 service recovery closure，不覆蓋原始 T+10 breach。
|
||||
|
||||
Windows 99 VMware task 已經由 `scripts/reboot-recovery/windows99-vmware-autostart-controlled-apply.ps1` 正式受控套用：DryRun、Apply、Verify 同一 receipt；task XML、舊 startup script、Windows Update policy 與 service start modes 都有 rollback。Evidence `C:\Wooo\Agent99\evidence\windows99-vmware-autostart-apply-20260711-173505.json` 回報 pre/post VMX process 都是 `5`、running guest rows `5`、process-aware duplicate-start guard true、沒有 host reboot、沒有 VM power change、沒有 secret read。這才是 Agent99 與重啟 SOP 的 runtime 整合證據；仍不取代下一次 600 秒 E2E drill。
|
||||
|
||||
@@ -113,7 +113,7 @@
|
||||
| 順序 | ID | 工作 | 狀態 | 驗收 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 5 | AG99-P0-005 | 模式專屬 verifier、真 rollback、cooldown/circuit-breaker | 待辦 | Recover/Perf/LoadShed/Harbor/AWOOOI/backup/provider 各有 post-condition；失敗不寫成功 cooldown |
|
||||
| 6 | AG99-P0-006 | Agent99 → AWOOOI/AwoooP/IWOOOS completion callback | **進行中；current P0** | incident/Run/Work Item/Verifier/recurrence 同一 trace 收斂；Telegram 可讀回報只從結果物件產生 |
|
||||
| 6 | AG99-P0-006 | Agent99 → AWOOOI/AwoooP/IWOOOS completion callback | **進行中；Agent99 runtime closure 已完成，AwoooP recurrence reconciliation 待 CD** | incident/Run/Work Item/Verifier/recurrence 同一 trace 收斂；Telegram 可讀回報只從結果物件產生 |
|
||||
| 7 | AG99-P0-007 | 全主機 10 分鐘 reboot state machine | **進行中；本輪 breach 後已 late recovery，L1/L2 runtime 已整合，L0/600 秒重測待完成** | L0 off-LAN detection/OOB power、L1 Agent99/VMware、L2 110 full-SOP 必須以同一 incident 閉環；首可用/全綠時間與 blocker/ETA 可查 |
|
||||
| 8 | AG99-P0-008 | 99 VMware/VMX/autostart/lock 與 Windows Update no-auto-reboot | **本次 boot runtime green；controlled apply/rollback/verifier 完成，待下次 reboot 驗證 startup 時序** | 110/188/112/120/121 VMX、autostart task、guest running 與 no-auto-reboot verifier green；111 由實體 Mac readback 驗證 |
|
||||
|
||||
@@ -206,6 +206,7 @@
|
||||
- 新版 live preflight 在 20:20:51 回報 9/9 tasks、relay=1、queue=0，required evidence `fresh=7/7`、`usable=7/7`、parse/content failure 都是 0；不再用檔案時間掩蓋 inbox JSON 解析失敗。
|
||||
- Production-principal Recover `agent99-recovery-20260711-202159-3404b5b3` 完成 5/5 VM、5/5 host、5/5 public 與 7/7 recovery phases；42.9 秒、coordinator verified、service-recovery only。TG receipt 是 `sent=1`、`visualSent=1`，實際 1200×760 圖卡顯示七階段時間軸。後續同狀態 run 以 `dedupe_window` suppressed 並保留 receipt。
|
||||
- `P0-006` commit `d140e35d3` 已新增 token-authenticated callback、idempotent channel event、AWOOOI operation receipts、AwoooP completed shadow run、durable DB readback，以及 99 pending/processed retry queue；payload 禁止 secret、raw log 與完整 evidence path。Focused tests 40 passed、ruff green；Windows `ValidateOnly` 通過 PowerShell parse/contract/synthetic/14-file staging，evidence 是 `agent99-deploy-20260711-204625.json`。Production callback route 仍為 HTTP 404，且 Administrator/Machine scope 僅證明 token presence=0、未讀值；API CD、production-principal auth verifier 與 live replay 尚未完成，所以維持 in progress。
|
||||
- 2026-07-14 runtime closure：Windows99 已部署 `568a72e7218fee8a588c31fe834d6751217640b0`，14/14 runtime、9/9 tasks、relay/live preflight green。29 筆 51 字元舊 callback 已原子遷移，最終 pending queue/callback 0、processed callback 40；最新真實 cold-start events 均為 resolved，callback `accepted/durableReadback=true` 且 operation receipt 2。AWOOOI public incident readback為 resolved，但 production recurrence 仍錯誤投影成 `run_completed_no_repair / automation_gap / needs_human=true`。本地 source `994d62b29` 補 API incident-id 防禦，`e090cf5a8` 只在同 callback 具 execution/resolved/verifier/source-event 四重證據且最新 stage resolved 時標記 `agent99_callback_verified`；40 focused tests green。兩個 commit 尚未 push/CD，因此 P0-006 只剩 read-model production deployment 與 visible recurrence verifier。
|
||||
|
||||
## 7. 工作項強制欄位與治理
|
||||
|
||||
@@ -222,4 +223,4 @@
|
||||
|
||||
## 8. 立即下一步
|
||||
|
||||
目前唯一 P0 是 `AG99-P0-006`。立即動作是提交 callback source，依單一 writer 協調取得 main push slot，讓 API 先經 Gitea CD 上線；再部署 99 callback client，重播一個受控 SRE alert，要求同一 callback/run/trace 在 AWOOOI operation log、AwoooP Run、IWOOOS owner plane、Agent99 local receipt 與 Telegram 都可讀回。後續固定依序：`P0-003 → P0-004 → P0-005 → P0-007 → P0-008 → P0-009 → P0-013 → P0-012 → P0-014 → P0-010 → P0-011`。main push 只在唯一 writer slot 進行；沒有 slot 時保留 local commit，不讓推送協調打斷 runtime 主線。
|
||||
目前唯一 P0 是 `AG99-P0-006`。Agent99 runtime、Telegram lifecycle、KM/RAG、AWOOOI callback durable DB receipts 與存量 replay 已完成；立即動作是將本地 `994d62b29 + e090cf5a8` 經唯一 writer 的 Gitea CD 投產，然後重讀 recurrence，要求 `agent99_callback_verified_group_total>=1`、該 work item closed、`needs_human=false`、automation gap 不再包含已驗證 Agent99 group。後續固定依序：`P0-003 → P0-004 → P0-005 → P0-007 → P0-008 → P0-009 → P0-013 → P0-012 → P0-014 → P0-010 → P0-011`。沒有 push slot 時保留 local commit並持續 runtime verifier，不讓協調訊息改排 current P0。
|
||||
|
||||
@@ -0,0 +1,274 @@
|
||||
[CmdletBinding()]
|
||||
param(
|
||||
[ValidateSet("Check", "Apply")]
|
||||
[string]$Mode = "Check",
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$TraceId,
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$RunId,
|
||||
[Parameter(Mandatory = $true)]
|
||||
[string]$WorkItemId,
|
||||
[string]$AgentRoot = "C:\Wooo\Agent99"
|
||||
)
|
||||
|
||||
$ErrorActionPreference = "Stop"
|
||||
$safeIdentityPattern = "^[A-Za-z0-9][A-Za-z0-9._:@+\-]{0,127}$"
|
||||
foreach ($identity in @($TraceId, $RunId, $WorkItemId)) {
|
||||
if ($identity -notmatch $safeIdentityPattern) {
|
||||
throw "controlled_identity_invalid"
|
||||
}
|
||||
}
|
||||
|
||||
$pendingDir = Join-Path $AgentRoot "state\completion-callbacks\pending"
|
||||
$processedCommandDir = Join-Path $AgentRoot "queue\processed"
|
||||
$evidenceDir = Join-Path $AgentRoot "evidence"
|
||||
$migrationRoot = Join-Path $AgentRoot "state\completion-callbacks\migration-backups"
|
||||
$lockPath = Join-Path $AgentRoot "state\completion-callbacks\incident-id-migration.lock"
|
||||
$script:MigrationLock = $null
|
||||
|
||||
function Get-Sha256Text {
|
||||
param([string]$Value)
|
||||
|
||||
$sha = [Security.Cryptography.SHA256]::Create()
|
||||
try {
|
||||
$bytes = [Text.Encoding]::UTF8.GetBytes($Value)
|
||||
return (($sha.ComputeHash($bytes) | ForEach-Object { $_.ToString("x2") }) -join "")
|
||||
} finally {
|
||||
$sha.Dispose()
|
||||
}
|
||||
}
|
||||
|
||||
function Get-Sha256File {
|
||||
param([string]$Path)
|
||||
return (Get-FileHash -Algorithm SHA256 -LiteralPath $Path).Hash.ToLowerInvariant()
|
||||
}
|
||||
|
||||
function Write-Utf8NoBom {
|
||||
param([string]$Path, [string]$Value)
|
||||
|
||||
$encoding = New-Object System.Text.UTF8Encoding($false)
|
||||
[IO.File]::WriteAllText($Path, $Value, $encoding)
|
||||
}
|
||||
|
||||
function Write-ImmutableJson {
|
||||
param([string]$Path, [object]$Value)
|
||||
|
||||
if (Test-Path -LiteralPath $Path) { throw "immutable_receipt_exists" }
|
||||
$temporaryPath = "$Path.tmp.$PID.$([Guid]::NewGuid().ToString('N'))"
|
||||
try {
|
||||
$json = $Value | ConvertTo-Json -Depth 12
|
||||
$null = $json | ConvertFrom-Json
|
||||
Write-Utf8NoBom $temporaryPath $json
|
||||
[IO.File]::Move($temporaryPath, $Path)
|
||||
} finally {
|
||||
if (Test-Path -LiteralPath $temporaryPath) {
|
||||
Remove-Item -LiteralPath $temporaryPath -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function Get-CallbackRows {
|
||||
$rows = @()
|
||||
$blockers = @()
|
||||
if (-not (Test-Path -LiteralPath $pendingDir -PathType Container)) {
|
||||
return [pscustomobject]@{ rows = @(); blockers = @() }
|
||||
}
|
||||
|
||||
foreach ($file in @(Get-ChildItem -LiteralPath $pendingDir -Filter "*.json" -File | Sort-Object Name)) {
|
||||
try {
|
||||
$payload = Get-Content -LiteralPath $file.FullName -Raw | ConvertFrom-Json
|
||||
$runIdValue = [string]$payload.run_id
|
||||
$priorAlertId = [string]$payload.alert_id
|
||||
if (-not $runIdValue -or $runIdValue -notmatch "^[A-Za-z0-9._-]{1,128}$") {
|
||||
throw "pending_run_id_invalid"
|
||||
}
|
||||
if (-not $priorAlertId) { throw "pending_alert_id_missing" }
|
||||
|
||||
$processedPath = Join-Path $processedCommandDir "processed-$runIdValue.json"
|
||||
if (-not (Test-Path -LiteralPath $processedPath -PathType Leaf)) {
|
||||
throw "processed_result_missing"
|
||||
}
|
||||
$processed = Get-Content -LiteralPath $processedPath -Raw | ConvertFrom-Json
|
||||
$canonicalIncidentId = ""
|
||||
if ($processed.PSObject.Properties["identity"] -and $processed.identity) {
|
||||
$canonicalIncidentId = [string]$processed.identity.incident_id
|
||||
}
|
||||
if (-not $canonicalIncidentId -and $processed.PSObject.Properties["incidentId"]) {
|
||||
$canonicalIncidentId = [string]$processed.incidentId
|
||||
}
|
||||
if (
|
||||
-not $canonicalIncidentId -or
|
||||
$canonicalIncidentId.Length -gt 30 -or
|
||||
$canonicalIncidentId -notmatch "^INC-[A-Za-z0-9-]+$"
|
||||
) {
|
||||
throw "canonical_incident_id_invalid"
|
||||
}
|
||||
|
||||
$rows += [pscustomobject]@{
|
||||
file = $file.Name
|
||||
processedFile = Split-Path -Leaf $processedPath
|
||||
runId = $runIdValue
|
||||
priorAlertIdLength = $priorAlertId.Length
|
||||
priorAlertIdSha256 = Get-Sha256Text $priorAlertId
|
||||
canonicalIncidentId = $canonicalIncidentId
|
||||
needsMigration = [bool]($priorAlertId -ne $canonicalIncidentId)
|
||||
beforeFileSha256 = Get-Sha256File $file.FullName
|
||||
}
|
||||
} catch {
|
||||
$blockers += [pscustomobject]@{
|
||||
file = $file.Name
|
||||
reason = $_.Exception.Message
|
||||
}
|
||||
}
|
||||
}
|
||||
[pscustomobject]@{ rows = @($rows); blockers = @($blockers) }
|
||||
}
|
||||
|
||||
$scan = Get-CallbackRows
|
||||
$candidateRows = @($scan.rows | Where-Object { $_.needsMigration })
|
||||
$identityToken = (Get-Sha256Text "$TraceId|$RunId|$WorkItemId").Substring(0, 20)
|
||||
$receiptPath = Join-Path $evidenceDir "agent99-callback-incident-id-migration-$identityToken.json"
|
||||
$backupDir = Join-Path $migrationRoot "backup-$identityToken"
|
||||
$baseReceipt = [ordered]@{
|
||||
schemaVersion = "agent99_callback_incident_id_migration_v1"
|
||||
status = "pending"
|
||||
observedAt = (Get-Date -Format o)
|
||||
mode = $Mode
|
||||
traceId = $TraceId
|
||||
runId = $RunId
|
||||
workItemId = $WorkItemId
|
||||
pendingCount = @($scan.rows).Count + @($scan.blockers).Count
|
||||
candidateCount = $candidateRows.Count
|
||||
unchangedCount = @($scan.rows | Where-Object { -not $_.needsMigration }).Count
|
||||
blockerCount = @($scan.blockers).Count
|
||||
blockers = @($scan.blockers)
|
||||
rows = @($scan.rows)
|
||||
receiptPath = $receiptPath
|
||||
backupPath = if ($Mode -eq "Apply") { $backupDir } else { $null }
|
||||
secretValueRead = $false
|
||||
privateKeyValueRead = $false
|
||||
tokenValueRead = $false
|
||||
rawExternalAlertIdLogged = $false
|
||||
remoteWritePerformed = $false
|
||||
pendingPayloadWritePerformed = $false
|
||||
rollbackAttempted = $false
|
||||
rollbackVerified = $false
|
||||
modifiedCount = 0
|
||||
completedAt = $null
|
||||
failureType = $null
|
||||
}
|
||||
|
||||
if ($Mode -eq "Check") {
|
||||
$baseReceipt.status = if (@($scan.blockers).Count -gt 0) { "blocked_check" } else { "check_ready" }
|
||||
$baseReceipt | ConvertTo-Json -Depth 12 -Compress
|
||||
if (@($scan.blockers).Count -gt 0) { exit 2 }
|
||||
exit 0
|
||||
}
|
||||
|
||||
if (@($scan.blockers).Count -gt 0) {
|
||||
$baseReceipt.status = "blocked_before_apply"
|
||||
$baseReceipt | ConvertTo-Json -Depth 12 -Compress
|
||||
exit 2
|
||||
}
|
||||
|
||||
New-Item -ItemType Directory -Force -Path $evidenceDir, $migrationRoot | Out-Null
|
||||
try {
|
||||
$script:MigrationLock = [IO.File]::Open(
|
||||
$lockPath,
|
||||
[IO.FileMode]::OpenOrCreate,
|
||||
[IO.FileAccess]::ReadWrite,
|
||||
[IO.FileShare]::None
|
||||
)
|
||||
} catch {
|
||||
$baseReceipt.status = "blocked_single_flight_lock_unavailable"
|
||||
$baseReceipt | ConvertTo-Json -Depth 12 -Compress
|
||||
exit 75
|
||||
}
|
||||
|
||||
$modifiedRows = @()
|
||||
try {
|
||||
if (Test-Path -LiteralPath $receiptPath) { throw "immutable_receipt_exists" }
|
||||
if (Test-Path -LiteralPath $backupDir) { throw "immutable_backup_path_exists" }
|
||||
New-Item -ItemType Directory -Path $backupDir | Out-Null
|
||||
$baseReceipt.remoteWritePerformed = $true
|
||||
|
||||
foreach ($row in $candidateRows) {
|
||||
$pendingPath = Join-Path $pendingDir $row.file
|
||||
$backupPath = Join-Path $backupDir $row.file
|
||||
$replaceBackupPath = Join-Path $backupDir "$($row.file).replace-original"
|
||||
if ((Get-Sha256File $pendingPath) -ne $row.beforeFileSha256) {
|
||||
throw "pending_payload_changed_after_check"
|
||||
}
|
||||
Copy-Item -LiteralPath $pendingPath -Destination $backupPath
|
||||
if ((Get-Sha256File $backupPath) -ne $row.beforeFileSha256) {
|
||||
throw "pending_payload_backup_mismatch"
|
||||
}
|
||||
|
||||
$payload = Get-Content -LiteralPath $pendingPath -Raw | ConvertFrom-Json
|
||||
$payload.alert_id = $row.canonicalIncidentId
|
||||
$temporaryPath = "$pendingPath.migrate.$PID.$([Guid]::NewGuid().ToString('N'))"
|
||||
try {
|
||||
$json = $payload | ConvertTo-Json -Depth 12
|
||||
Write-Utf8NoBom $temporaryPath $json
|
||||
$candidate = Get-Content -LiteralPath $temporaryPath -Raw | ConvertFrom-Json
|
||||
if ([string]$candidate.alert_id -ne $row.canonicalIncidentId) {
|
||||
throw "candidate_post_condition_failed"
|
||||
}
|
||||
[IO.File]::Replace($temporaryPath, $pendingPath, $replaceBackupPath, $true)
|
||||
if ((Get-Sha256File $replaceBackupPath) -ne $row.beforeFileSha256) {
|
||||
throw "atomic_replace_backup_mismatch"
|
||||
}
|
||||
$modifiedRows += $row
|
||||
} finally {
|
||||
if (Test-Path -LiteralPath $temporaryPath) {
|
||||
Remove-Item -LiteralPath $temporaryPath -Force -ErrorAction SilentlyContinue
|
||||
}
|
||||
}
|
||||
$baseReceipt.pendingPayloadWritePerformed = $true
|
||||
$postPayload = Get-Content -LiteralPath $pendingPath -Raw | ConvertFrom-Json
|
||||
if ([string]$postPayload.alert_id -ne $row.canonicalIncidentId) {
|
||||
throw "post_migration_readback_failed"
|
||||
}
|
||||
}
|
||||
|
||||
$baseReceipt.status = if ($candidateRows.Count -gt 0) { "migrated_verified" } else { "no_change_verified" }
|
||||
$baseReceipt.modifiedCount = $modifiedRows.Count
|
||||
$baseReceipt.completedAt = (Get-Date -Format o)
|
||||
Write-ImmutableJson $receiptPath $baseReceipt
|
||||
$baseReceipt | ConvertTo-Json -Depth 12 -Compress
|
||||
exit 0
|
||||
} catch {
|
||||
$failureType = $_.Exception.Message
|
||||
$baseReceipt.rollbackAttempted = [bool]($modifiedRows.Count -gt 0)
|
||||
$rollbackOk = $true
|
||||
foreach ($row in $modifiedRows) {
|
||||
try {
|
||||
$pendingPath = Join-Path $pendingDir $row.file
|
||||
$backupPath = Join-Path $backupDir $row.file
|
||||
$failedReplacementPath = Join-Path $backupDir "$($row.file).failed-migrated"
|
||||
$temporaryPath = "$pendingPath.rollback.$PID.$([Guid]::NewGuid().ToString('N'))"
|
||||
Copy-Item -LiteralPath $backupPath -Destination $temporaryPath
|
||||
[IO.File]::Replace($temporaryPath, $pendingPath, $failedReplacementPath, $true)
|
||||
if ((Get-Sha256File $pendingPath) -ne $row.beforeFileSha256) {
|
||||
$rollbackOk = $false
|
||||
}
|
||||
} catch {
|
||||
$rollbackOk = $false
|
||||
}
|
||||
}
|
||||
$baseReceipt.rollbackVerified = [bool]($baseReceipt.rollbackAttempted -and $rollbackOk)
|
||||
$baseReceipt.status = if ($baseReceipt.rollbackVerified) { "rolled_back_verified" } else { "migration_failed" }
|
||||
$baseReceipt.failureType = $failureType
|
||||
$baseReceipt.modifiedCount = $modifiedRows.Count
|
||||
$baseReceipt.completedAt = (Get-Date -Format o)
|
||||
if (-not (Test-Path -LiteralPath $receiptPath)) {
|
||||
Write-ImmutableJson $receiptPath $baseReceipt
|
||||
}
|
||||
$baseReceipt | ConvertTo-Json -Depth 12 -Compress
|
||||
exit 2
|
||||
} finally {
|
||||
if ($script:MigrationLock) {
|
||||
$script:MigrationLock.Dispose()
|
||||
$script:MigrationLock = $null
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[3]
|
||||
SCRIPT = ROOT / "scripts/reboot-recovery/agent99-completion-callback-incident-id-migration.ps1"
|
||||
|
||||
|
||||
def test_migration_requires_controlled_identity_and_supports_check_apply() -> None:
|
||||
text = SCRIPT.read_text(encoding="utf-8")
|
||||
|
||||
assert '[ValidateSet("Check", "Apply")]' in text
|
||||
assert "controlled_identity_invalid" in text
|
||||
assert '"check_ready"' in text
|
||||
assert '"migrated_verified"' in text
|
||||
assert '"rolled_back_verified"' in text
|
||||
|
||||
|
||||
def test_migration_uses_canonical_processed_identity_and_bounded_incident_id() -> None:
|
||||
text = SCRIPT.read_text(encoding="utf-8")
|
||||
|
||||
assert 'processed.identity.incident_id' in text
|
||||
assert '$canonicalIncidentId.Length -gt 30' in text
|
||||
assert '"^INC-[A-Za-z0-9-]+$"' in text
|
||||
assert 'rawExternalAlertIdLogged = $false' in text
|
||||
assert 'priorAlertIdSha256 = Get-Sha256Text $priorAlertId' in text
|
||||
|
||||
|
||||
def test_migration_is_atomic_backed_up_and_fail_closed() -> None:
|
||||
text = SCRIPT.read_text(encoding="utf-8")
|
||||
|
||||
assert '[IO.File]::Replace($temporaryPath, $pendingPath, $replaceBackupPath, $true)' in text
|
||||
assert "atomic_replace_backup_mismatch" in text
|
||||
assert 'pending_payload_backup_mismatch' in text
|
||||
assert 'pending_payload_changed_after_check' in text
|
||||
assert 'immutable_receipt_exists' in text
|
||||
assert 'rollbackVerified' in text
|
||||
assert 'secretValueRead = $false' in text
|
||||
Reference in New Issue
Block a user