fix(agent99): close host110 backup commit races

2026-07-19 01:08:45 +08:00
parent 8f11ef3362
commit d1797a1c87
5 changed files with 1282 additions and 42 deletions
--- a/docs/proposals/AWOOOI-AG99-H110-BACKUP-RUNTIME-V10.md
+++ b/docs/proposals/AWOOOI-AG99-H110-BACKUP-RUNTIME-V10.md
@@ -0,0 +1,217 @@
+# AWOOOI Agent99 Host110 Backup Runtime V10
+
+Status: proposed; central review and explicit owner approval are required before
+any production apply.
+
+This artifact supersedes V2 through V9. Every V9 identity below is void for
+approval and remains audit evidence only:
+
+- branch/ref: `codex/host110-backup-runtime-v9-20260718`
+- candidate/revision: `8f11ef3362a8d7f66a46308b58fd142b0c91a8bc`
+- artifact SHA-256: `929978eb3da15b04791a8614e6d5ee504ae2a4c2f205b694307a7bad7713bcad`
+- exact diff SHA-256: `8085d49450b74e7c43577ba118cbd74d3a2e3dd2d22eeba90ce749d56c72c0f2`
+- every other V9 proposal, content, revision, and derived hash
+
+No V9 hash, ref, review, receipt, or test grants production execution
+authority.
+
+## Exact Scope
+
+- Agent99 runtime bundle: 19 files, guarded by one producer/consumer parity
+  test.
+- Host110 backup payload: 18 ordered files, comprising 17 backup scripts and
+  `backup-health-textfile-exporter.py`.
+- Host110 remote stage: 21 unique basenames, comprising the ordered 18-file
+  payload, fixed executor, independent verifier, and `manifest.json`.
+- Fixed target: `wooo@192.168.0.110`, dispatched by Windows99 Agent99 from an
+  exact Gitea revision and digest manifest.
+- The candidate includes a 10-line runtime-lock addition to
+  `scripts/backup/gitea-full-backup-restore-drill.sh`. A future drill invocation
+  acquires the shared Host110 backup-runtime lock before its pre-existing drill
+  behavior. Applying this candidate replaces the script but does not invoke the
+  drill or any container lifecycle.
+- `scripts/backup/backup-gitea.sh` has zero diff from the production base. No
+  Gitea primary stop/start or backup-container creation is part of this
+  candidate apply.
+- V7, V8, and V9 proposal documents remain tracked audit artifacts and are not
+  runtime authority.
+- `agent99-host110-broker-contract-replay.ps1` is a no-write Windows99 test
+  asset; it is not part of the 19-file Agent99 runtime or 18-file Host110
+  payload.
+
+## V9 Defects Closed
+
+### Signal-Safe Commit Boundary
+
+The executor enters a signal-deferring critical section before publishing the
+payload receipt. `INT`, `HUP`, and `TERM` are recorded without exiting until the
+durable payload-receipt readback and `APPLY_COMPLETE=1` commit state are both
+established. A deferred signal can never enter the pre-commit EXIT rollback
+path after the payload receipt exists.
+
+After commit, exact stage and rollback-prestate cleanup still run. If cleanup
+is complete, a deferred signal produces the non-success terminal
+`committed_signal_deferred`, a durable terminal receipt, the corresponding
+129/130/143 exit, and an applied payload. TERM, HUP, and INT are replayed in the
+exact receipt-to-commit window.
+
+After cleanup, terminal publication uses a bounded signal mask. Signals captured
+before that boundary remain part of the non-success terminal; signals delivered
+during the immutable terminal writer cannot mutate its already-final state.
+Terminal-writer TERM, HUP, and INT replays require receipt, stdout, payload, and
+exit to remain consistent.
+
+### Payload And Terminal Receipts
+
+The immutable payload receipt has status `payload_verified`; it proves exact
+payload, independent verifier, durable receipt readback, and absence of
+run-owned destination temporaries. It does not claim internal stage or rollback
+prestate cleanup.
+
+A separate immutable terminal receipt is the authority for overall success.
+Only status `verified` with `stageCleanupVerified=true`,
+`rollbackPrestateCleanupVerified=true`, no deferred signal, and exit code zero
+may yield executor `ok:true`. Each successful cleanup is absence-read back and
+its parent directory is fsynced before terminal publication.
+
+If either cleanup fails, the payload remains committed, exact residue is
+preserved, terminal status is `cleanup_pending`, executor output is
+`ok:false`, and exit is nonzero. A payload receipt cannot be used as a
+zero-residue or broker-success shortcut. The Windows99 broker validates every
+status, payload, rollback, residue-scope, cleanup, signal, exit, receipt-path,
+and independent-verifier field fail-closed.
+Committed non-success outputs are parsed and retained in broker evidence before
+the broker fails overall; `cleanup_pending`, `committed_signal_deferred`, and
+`terminal_receipt_failed` can never be collapsed into an ambiguous generic
+transport failure.
+
+### Existing Transaction Guarantees Retained
+
+- Producer and executor payload basenames are exact-equal in count, set, and
+  order: 18 payload files and 21 unique staged files.
+- Receipt write, hard-link, fsync, and readback failures roll back exact content
+  and metadata without publishing success.
+- Terminal-receipt write, hard-link, fsync, and readback failures keep the
+  verified payload, publish no terminal success, emit structured committed
+  failure evidence with `terminalReceiptPublished=false`, and exit nonzero.
+- Apply and rollback destination temporaries are run-owned, tracked, removed,
+  and absence-read back.
+- Rollback attempted, performed, verified, and residue truth remain separate;
+  partial or unknown rollback terminates `rollback_unverified`.
+- Fault hooks require both the explicit test flag and a transformed
+  non-production destination; canonical `/backup/scripts` cannot enable them.
+
+## Apply Effects
+
+- Windows99 Agent99 runtime promotion writes the 19 runtime files plus bounded
+  manifest and receipt state.
+- Host110 Apply replaces the ordered 18 payload paths through one exclusive,
+  digest-bound filesystem transaction, and writes payload/terminal receipts
+  below `/backup/status`.
+- The deployed restore-drill script gains shared-lock behavior. The apply does
+  not execute that script.
+- Apply does not run a backup, invoke a Gitea container lifecycle, sync Google
+  Drive, prune snapshots, restart a VM or service, or change a database,
+  network, firewall, credential, or secret.
+
+## Reproducible Validation
+
+Focused replay and contract suite:
+
+```sh
+pytest -q \
+  scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
+  scripts/backup/tests/test_verify_offsite_full_sync_runtime.py \
+  scripts/ops/tests/test_agent99_signoz_metadata_executor_runtime_entrypoint.py \
+  scripts/ops/tests/test_db_bounded_executor_runtime_entrypoint.py \
+  scripts/reboot-recovery/tests/test_agent99_live_preflight_decision.py \
+  scripts/reboot-recovery/tests/test_agent99_remote_atomic_deploy_transport_contract.py \
+  scripts/reboot-recovery/tests/test_host112_manager_recovery_contract.py
+```
+
+Exact signal and post-commit cleanup replays:
+
+```sh
+pytest -vv \
+  scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
+  -k 'commit_window_signal or terminal_publication or terminal_receipt_fault or postcommit_cleanup'
+```
+
+Static, syntax, and repository checks:
+
+```sh
+ruff check scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py
+bash -n scripts/backup/host110-backup-runtime-executor.sh
+git diff --check
+PATH=/Users/ogt/.pyenv/shims:$PATH bash scripts/ops/ansible-validate.sh
+```
+
+`ansible-validate.sh` parses
+`ops/reboot-recovery/full-stack-backup-baseline.yml` as YAML metadata. It runs
+`ansible-playbook --syntax-check` only for the actual playbooks listed by the
+script; the metadata mapping is not represented as a playbook.
+
+Windows99 no-file-write PowerShell parser check:
+
+```sh
+PARSER_COMMAND='$source=[Console]::In.ReadToEnd(); $tokens=$null; $errors=$null; [System.Management.Automation.Language.Parser]::ParseInput($source,[ref]$tokens,[ref]$errors) | Out-Null; [pscustomobject]@{ok=($errors.Count -eq 0); errorCount=$errors.Count; parser="WindowsPowerShell"} | ConvertTo-Json -Compress; if ($errors.Count -ne 0) { exit 1 }'
+ENCODED=$(printf '%s' "$PARSER_COMMAND" | iconv -f UTF-8 -t UTF-16LE | base64 | tr -d '\n')
+ssh -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=5 \
+  -o ConnectionAttempts=1 Administrator@192.168.0.99 \
+  "powershell.exe -NoProfile -NonInteractive -EncodedCommand $ENCODED" \
+  < agent99-host110-backup-runtime-broker.ps1
+```
+
+Windows99 actual-function missing-field/type contract replay:
+
+```sh
+WRAPPER='$script=[Console]::In.ReadToEnd(); & ([ScriptBlock]::Create($script))'
+WRAPPER_ENCODED=$(printf '%s' "$WRAPPER" | iconv -f UTF-8 -t UTF-16LE | base64 | tr -d '\n')
+BROKER_GZIP_BASE64=$(gzip -c agent99-host110-backup-runtime-broker.ps1 | base64 | tr -d '\n')
+awk -v payload="$BROKER_GZIP_BASE64" \
+  'BEGIN { print "$BrokerGzipBase64 = \"" payload "\"" } { print }' \
+  scripts/reboot-recovery/tests/agent99-host110-broker-contract-replay.ps1 |
+  ssh -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=5 \
+    -o ConnectionAttempts=1 Administrator@192.168.0.99 \
+    "powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -EncodedCommand $WRAPPER_ENCODED"
+```
+
+The replay loads the candidate broker's actual function AST, accepts the valid
+success and three committed non-success terminals, then removes every required
+field and injects wrong JSON types. Expected readback is 34 missing-field and 6
+type rejections. It performs no remote file write.
+
+## Exact Diff Serialization
+
+Substitute the V10 Gitea live-ref SHA for `<candidate_git_sha>` and hash the
+exact stdout bytes:
+
+```sh
+git diff --binary --full-index \
+  e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3..<candidate_git_sha> |
+  shasum -a 256
+```
+
+Central review must independently bind the live feature ref and candidate Git
+SHA, this tracked artifact path and bytes hash, the serialized diff hash/path
+count/line counts, the 19/18/21 count contracts, and the exact restore-drill and
+`backup-gitea.sh` diffs.
+
+## Rollback
+
+- Source rollback reverts the exact V10 diff to
+  `e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3` without force-pushing.
+- Agent99 rollback restores the prior runtime files and manifest and verifies
+  prior hashes independently.
+- Before payload commit, Host110 rollback restores all prior payload content,
+  SHA-256, uid, gid, and mode under the exclusive lock and proves no run-owned
+  destination temporary remains.
+- After payload commit, the executor never rolls the payload back merely due to
+  a signal or cleanup error. It preserves exact cleanup evidence and reports a
+  non-success terminal. Any later cleanup-only remediation is a separate
+  controlled action scoped to the run-specific stage/prestate paths and may not
+  rewrite the verified payload.
+- Source rollback restores the restore-drill script's prior unlocked behavior.
+  It does not invoke the drill or any container lifecycle.
+- Any canonical, metadata, receipt, signal, cleanup, or residue mismatch fails
+  closed and cannot produce broker `verified`.