Compare commits
2 Commits
codex/host
...
codex/host
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2c2dc086b0 | ||
|
|
d1797a1c87 |
File diff suppressed because it is too large
Load Diff
217
docs/proposals/AWOOOI-AG99-H110-BACKUP-RUNTIME-V10.md
Normal file
217
docs/proposals/AWOOOI-AG99-H110-BACKUP-RUNTIME-V10.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# AWOOOI Agent99 Host110 Backup Runtime V10
|
||||
|
||||
Status: proposed; central review and explicit owner approval are required before
|
||||
any production apply.
|
||||
|
||||
This artifact supersedes V2 through V9. Every V9 identity below is void for
|
||||
approval and remains audit evidence only:
|
||||
|
||||
- branch/ref: `codex/host110-backup-runtime-v9-20260718`
|
||||
- candidate/revision: `8f11ef3362a8d7f66a46308b58fd142b0c91a8bc`
|
||||
- artifact SHA-256: `929978eb3da15b04791a8614e6d5ee504ae2a4c2f205b694307a7bad7713bcad`
|
||||
- exact diff SHA-256: `8085d49450b74e7c43577ba118cbd74d3a2e3dd2d22eeba90ce749d56c72c0f2`
|
||||
- every other V9 proposal, content, revision, and derived hash
|
||||
|
||||
No V9 hash, ref, review, receipt, or test grants production execution
|
||||
authority.
|
||||
|
||||
## Exact Scope
|
||||
|
||||
- Agent99 runtime bundle: 19 files, guarded by one producer/consumer parity
|
||||
test.
|
||||
- Host110 backup payload: 18 ordered files, comprising 17 backup scripts and
|
||||
`backup-health-textfile-exporter.py`.
|
||||
- Host110 remote stage: 21 unique basenames, comprising the ordered 18-file
|
||||
payload, fixed executor, independent verifier, and `manifest.json`.
|
||||
- Fixed target: `wooo@192.168.0.110`, dispatched by Windows99 Agent99 from an
|
||||
exact Gitea revision and digest manifest.
|
||||
- The candidate includes a 10-line runtime-lock addition to
|
||||
`scripts/backup/gitea-full-backup-restore-drill.sh`. A future drill invocation
|
||||
acquires the shared Host110 backup-runtime lock before its pre-existing drill
|
||||
behavior. Applying this candidate replaces the script but does not invoke the
|
||||
drill or any container lifecycle.
|
||||
- `scripts/backup/backup-gitea.sh` has zero diff from the production base. No
|
||||
Gitea primary stop/start or backup-container creation is part of this
|
||||
candidate apply.
|
||||
- V7, V8, and V9 proposal documents remain tracked audit artifacts and are not
|
||||
runtime authority.
|
||||
- `agent99-host110-broker-contract-replay.ps1` is a no-write Windows99 test
|
||||
asset; it is not part of the 19-file Agent99 runtime or 18-file Host110
|
||||
payload.
|
||||
|
||||
## V9 Defects Closed
|
||||
|
||||
### Signal-Safe Commit Boundary
|
||||
|
||||
The executor enters a signal-deferring critical section before publishing the
|
||||
payload receipt. `INT`, `HUP`, and `TERM` are recorded without exiting until the
|
||||
durable payload-receipt readback and `APPLY_COMPLETE=1` commit state are both
|
||||
established. A deferred signal can never enter the pre-commit EXIT rollback
|
||||
path after the payload receipt exists.
|
||||
|
||||
After commit, exact stage and rollback-prestate cleanup still run. If cleanup
|
||||
is complete, a deferred signal produces the non-success terminal
|
||||
`committed_signal_deferred`, a durable terminal receipt, the corresponding
|
||||
129/130/143 exit, and an applied payload. TERM, HUP, and INT are replayed in the
|
||||
exact receipt-to-commit window.
|
||||
|
||||
After cleanup, terminal publication uses a bounded signal mask. Signals captured
|
||||
before that boundary remain part of the non-success terminal; signals delivered
|
||||
during the immutable terminal writer cannot mutate its already-final state.
|
||||
Terminal-writer TERM, HUP, and INT replays require receipt, stdout, payload, and
|
||||
exit to remain consistent.
|
||||
|
||||
### Payload And Terminal Receipts
|
||||
|
||||
The immutable payload receipt has status `payload_verified`; it proves exact
|
||||
payload, independent verifier, durable receipt readback, and absence of
|
||||
run-owned destination temporaries. It does not claim internal stage or rollback
|
||||
prestate cleanup.
|
||||
|
||||
A separate immutable terminal receipt is the authority for overall success.
|
||||
Only status `verified` with `stageCleanupVerified=true`,
|
||||
`rollbackPrestateCleanupVerified=true`, no deferred signal, and exit code zero
|
||||
may yield executor `ok:true`. Each successful cleanup is absence-read back and
|
||||
its parent directory is fsynced before terminal publication.
|
||||
|
||||
If either cleanup fails, the payload remains committed, exact residue is
|
||||
preserved, terminal status is `cleanup_pending`, executor output is
|
||||
`ok:false`, and exit is nonzero. A payload receipt cannot be used as a
|
||||
zero-residue or broker-success shortcut. The Windows99 broker validates every
|
||||
status, payload, rollback, residue-scope, cleanup, signal, exit, receipt-path,
|
||||
and independent-verifier field fail-closed.
|
||||
Committed non-success outputs are parsed and retained in broker evidence before
|
||||
the broker fails overall; `cleanup_pending`, `committed_signal_deferred`, and
|
||||
`terminal_receipt_failed` can never be collapsed into an ambiguous generic
|
||||
transport failure.
|
||||
|
||||
### Existing Transaction Guarantees Retained
|
||||
|
||||
- Producer and executor payload basenames are exact-equal in count, set, and
|
||||
order: 18 payload files and 21 unique staged files.
|
||||
- Receipt write, hard-link, fsync, and readback failures roll back exact content
|
||||
and metadata without publishing success.
|
||||
- Terminal-receipt write, hard-link, fsync, and readback failures keep the
|
||||
verified payload, publish no terminal success, emit structured committed
|
||||
failure evidence with `terminalReceiptPublished=false`, and exit nonzero.
|
||||
- Apply and rollback destination temporaries are run-owned, tracked, removed,
|
||||
and absence-read back.
|
||||
- Rollback attempted, performed, verified, and residue truth remain separate;
|
||||
partial or unknown rollback terminates `rollback_unverified`.
|
||||
- Fault hooks require both the explicit test flag and a transformed
|
||||
non-production destination; canonical `/backup/scripts` cannot enable them.
|
||||
|
||||
## Apply Effects
|
||||
|
||||
- Windows99 Agent99 runtime promotion writes the 19 runtime files plus bounded
|
||||
manifest and receipt state.
|
||||
- Host110 Apply replaces the ordered 18 payload paths through one exclusive,
|
||||
digest-bound filesystem transaction, and writes payload/terminal receipts
|
||||
below `/backup/status`.
|
||||
- The deployed restore-drill script gains shared-lock behavior. The apply does
|
||||
not execute that script.
|
||||
- Apply does not run a backup, invoke a Gitea container lifecycle, sync Google
|
||||
Drive, prune snapshots, restart a VM or service, or change a database,
|
||||
network, firewall, credential, or secret.
|
||||
|
||||
## Reproducible Validation
|
||||
|
||||
Focused replay and contract suite:
|
||||
|
||||
```sh
|
||||
pytest -q \
|
||||
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
|
||||
scripts/backup/tests/test_verify_offsite_full_sync_runtime.py \
|
||||
scripts/ops/tests/test_agent99_signoz_metadata_executor_runtime_entrypoint.py \
|
||||
scripts/ops/tests/test_db_bounded_executor_runtime_entrypoint.py \
|
||||
scripts/reboot-recovery/tests/test_agent99_live_preflight_decision.py \
|
||||
scripts/reboot-recovery/tests/test_agent99_remote_atomic_deploy_transport_contract.py \
|
||||
scripts/reboot-recovery/tests/test_host112_manager_recovery_contract.py
|
||||
```
|
||||
|
||||
Exact signal and post-commit cleanup replays:
|
||||
|
||||
```sh
|
||||
pytest -vv \
|
||||
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
|
||||
-k 'commit_window_signal or terminal_publication or terminal_receipt_fault or postcommit_cleanup'
|
||||
```
|
||||
|
||||
Static, syntax, and repository checks:
|
||||
|
||||
```sh
|
||||
ruff check scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py
|
||||
bash -n scripts/backup/host110-backup-runtime-executor.sh
|
||||
git diff --check
|
||||
PATH=/Users/ogt/.pyenv/shims:$PATH bash scripts/ops/ansible-validate.sh
|
||||
```
|
||||
|
||||
`ansible-validate.sh` parses
|
||||
`ops/reboot-recovery/full-stack-backup-baseline.yml` as YAML metadata. It runs
|
||||
`ansible-playbook --syntax-check` only for the actual playbooks listed by the
|
||||
script; the metadata mapping is not represented as a playbook.
|
||||
|
||||
Windows99 no-file-write PowerShell parser check:
|
||||
|
||||
```sh
|
||||
PARSER_COMMAND='$source=[Console]::In.ReadToEnd(); $tokens=$null; $errors=$null; [System.Management.Automation.Language.Parser]::ParseInput($source,[ref]$tokens,[ref]$errors) | Out-Null; [pscustomobject]@{ok=($errors.Count -eq 0); errorCount=$errors.Count; parser="WindowsPowerShell"} | ConvertTo-Json -Compress; if ($errors.Count -ne 0) { exit 1 }'
|
||||
ENCODED=$(printf '%s' "$PARSER_COMMAND" | iconv -f UTF-8 -t UTF-16LE | base64 | tr -d '\n')
|
||||
ssh -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=5 \
|
||||
-o ConnectionAttempts=1 Administrator@192.168.0.99 \
|
||||
"powershell.exe -NoProfile -NonInteractive -EncodedCommand $ENCODED" \
|
||||
< agent99-host110-backup-runtime-broker.ps1
|
||||
```
|
||||
|
||||
Windows99 actual-function missing-field/type contract replay:
|
||||
|
||||
```sh
|
||||
WRAPPER='$script=[Console]::In.ReadToEnd(); & ([ScriptBlock]::Create($script))'
|
||||
WRAPPER_ENCODED=$(printf '%s' "$WRAPPER" | iconv -f UTF-8 -t UTF-16LE | base64 | tr -d '\n')
|
||||
BROKER_GZIP_BASE64=$(gzip -c agent99-host110-backup-runtime-broker.ps1 | base64 | tr -d '\n')
|
||||
awk -v payload="$BROKER_GZIP_BASE64" \
|
||||
'BEGIN { print "$BrokerGzipBase64 = \"" payload "\"" } { print }' \
|
||||
scripts/reboot-recovery/tests/agent99-host110-broker-contract-replay.ps1 |
|
||||
ssh -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=5 \
|
||||
-o ConnectionAttempts=1 Administrator@192.168.0.99 \
|
||||
"powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -EncodedCommand $WRAPPER_ENCODED"
|
||||
```
|
||||
|
||||
The replay loads the candidate broker's actual function AST, accepts the valid
|
||||
success and three committed non-success terminals, then removes every required
|
||||
field and injects wrong JSON types. Expected readback is 34 missing-field and 6
|
||||
type rejections. It performs no remote file write.
|
||||
|
||||
## Exact Diff Serialization
|
||||
|
||||
Substitute the V10 Gitea live-ref SHA for `<candidate_git_sha>` and hash the
|
||||
exact stdout bytes:
|
||||
|
||||
```sh
|
||||
git diff --binary --full-index \
|
||||
e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3..<candidate_git_sha> |
|
||||
shasum -a 256
|
||||
```
|
||||
|
||||
Central review must independently bind the live feature ref and candidate Git
|
||||
SHA, this tracked artifact path and bytes hash, the serialized diff hash/path
|
||||
count/line counts, the 19/18/21 count contracts, and the exact restore-drill and
|
||||
`backup-gitea.sh` diffs.
|
||||
|
||||
## Rollback
|
||||
|
||||
- Source rollback reverts the exact V10 diff to
|
||||
`e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3` without force-pushing.
|
||||
- Agent99 rollback restores the prior runtime files and manifest and verifies
|
||||
prior hashes independently.
|
||||
- Before payload commit, Host110 rollback restores all prior payload content,
|
||||
SHA-256, uid, gid, and mode under the exclusive lock and proves no run-owned
|
||||
destination temporary remains.
|
||||
- After payload commit, the executor never rolls the payload back merely due to
|
||||
a signal or cleanup error. It preserves exact cleanup evidence and reports a
|
||||
non-success terminal. Any later cleanup-only remediation is a separate
|
||||
controlled action scoped to the run-specific stage/prestate paths and may not
|
||||
rewrite the verified payload.
|
||||
- Source rollback restores the restore-drill script's prior unlocked behavior.
|
||||
It does not invoke the drill or any container lifecycle.
|
||||
- Any canonical, metadata, receipt, signal, cleanup, or residue mismatch fails
|
||||
closed and cannot produce broker `verified`.
|
||||
202
docs/proposals/AWOOOI-AG99-H110-BACKUP-RUNTIME-V11.md
Normal file
202
docs/proposals/AWOOOI-AG99-H110-BACKUP-RUNTIME-V11.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# AWOOOI Agent99 Host110 Backup Runtime V11
|
||||
|
||||
Status: proposed. Central review and explicit owner approval are required before
|
||||
any production apply. This candidate must not be pushed to `main`, sent through
|
||||
CD, or applied to Windows99 or Host110 while it remains proposed.
|
||||
|
||||
This artifact supersedes V10. Every V10 identity below is void for approval and
|
||||
remains audit evidence only:
|
||||
|
||||
- branch/ref: `codex/host110-backup-runtime-v10-20260718`
|
||||
- candidate/revision: `d1797a1c8766722ae00ac14e20fed00f4e3f3a5b`
|
||||
- artifact SHA-256: `eb48661bbecc4e169111879bb18dfab63be29878d5f5c3651d9917f81f195d28`
|
||||
- exact diff SHA-256: `f7efb98d5f91fa6899713b692c8bb015fcb6ae6dba4291cc6ae429290072964e`
|
||||
- every other V10 proposal, content, revision, review, and derived hash
|
||||
|
||||
No V10 hash, ref, review, receipt, or test grants production execution
|
||||
authority.
|
||||
|
||||
## Exact Scope
|
||||
|
||||
- Agent99 runtime bundle: 19 files, guarded by one producer/consumer parity
|
||||
test.
|
||||
- Host110 backup payload: 18 ordered files, comprising 17 backup scripts and
|
||||
`backup-health-textfile-exporter.py`.
|
||||
- Host110 remote stage: 21 unique basenames, comprising the ordered 18-file
|
||||
payload, fixed executor, independent verifier, and `manifest.json`.
|
||||
- Fixed target: `wooo@192.168.0.110`, dispatched only by the Windows99 Agent99
|
||||
typed broker from an exact Gitea revision and digest-bound manifest.
|
||||
- V11 changes the Windows99 broker, three shell shared-lock entrypoints, the
|
||||
Host188 archive verifier's shared-lock acquisition, and their no-write replay
|
||||
tests.
|
||||
- `scripts/backup/backup-gitea.sh` has zero diff from the production base. No
|
||||
Gitea primary stop/start or backup-container creation is part of this apply.
|
||||
- The restore drill and offsite gate gain only canonical shared-lock behavior.
|
||||
Applying this candidate replaces those scripts but does not invoke a drill,
|
||||
run a backup, or start an offsite sync.
|
||||
- Prior proposal documents remain tracked audit artifacts and are not runtime
|
||||
authority.
|
||||
|
||||
## V10 Defects Closed
|
||||
|
||||
### Persisted Broker Evidence Is Exact And Fail-Closed
|
||||
|
||||
Before reusing a run-bound broker receipt, the broker now validates exact key
|
||||
sets, required native JSON types, schema version, run ID, source revision, mode,
|
||||
target, evidence path binding, status, error, cleanup, and nested Apply/Verify
|
||||
contracts. Missing, extra, wrong-type, wrong-mode, wrong-run, conflicting, or
|
||||
malformed evidence fails closed before any replay decision.
|
||||
|
||||
A non-empty string such as `"false"` can no longer be cast to a truthy Boolean.
|
||||
An existing failure remains a failure and cannot be upgraded to success merely
|
||||
because it is parseable or shares a source revision.
|
||||
|
||||
### Mandatory Post-Commit Transport Reconciliation
|
||||
|
||||
If Apply loses SSH transport or times out, the broker performs a read-only,
|
||||
exact-RunId receipt readback under `/backup/status` and an independent
|
||||
destination verifier readback. It never blindly retries Apply and never
|
||||
collapses a post-commit state into a generic precommit failure.
|
||||
|
||||
The reconciliation states are explicit and non-success:
|
||||
|
||||
- `committed_verified_terminal_missing`
|
||||
- `committed_verified_transport_lost`
|
||||
- `committed_cleanup_pending_transport_lost`
|
||||
- `committed_signal_deferred_transport_lost`
|
||||
- `committed_unknown`
|
||||
|
||||
Every state records `retryApplyAllowed=false`. Receipt schema, native types,
|
||||
source/run/head identity, payload paths and counts, verifier identity, cleanup,
|
||||
signal, and terminal semantics are validated before classification. The
|
||||
readback performs no remote write and reads no secret value.
|
||||
|
||||
### Canonical Shared Lock And Nested Re-entry
|
||||
|
||||
Production backup entrypoints canonicalize absolute, relative, and symlink
|
||||
invocation paths before deciding whether the Host110 runtime lock applies.
|
||||
Unresolvable production identity fails closed.
|
||||
|
||||
For nested calls, an inherited FD 197 is accepted only when its canonical
|
||||
target is the exact runtime lock. Linux uses `/proc/$$/fd/197`; the bounded
|
||||
non-Linux fallback resolves the descriptor path with `F_GETPATH`. The entrypoint
|
||||
then requests the shared lock on the same descriptor. It does not close and
|
||||
reopen a valid inherited descriptor, so there is no release/reacquire gap.
|
||||
|
||||
If FD 197 is absent, the entrypoint opens the lock and acquires the shared side.
|
||||
If FD 197 exists but points elsewhere or cannot be resolved, execution fails
|
||||
closed. `BACKUP_RUNTIME_SHARED_LOCK_HELD` and all other external environment
|
||||
values grant no lock authority.
|
||||
|
||||
The Host188 archive verifier follows the same contract: exact inherited FD 197
|
||||
is duplicated from the same open file description, wrong identity fails
|
||||
closed, and a forged environment value cannot bypass an active exclusive lock.
|
||||
|
||||
## Apply Effects
|
||||
|
||||
- Windows99 Agent99 runtime promotion writes the 19 runtime files plus bounded
|
||||
manifest and receipt state.
|
||||
- Host110 Apply replaces the ordered 18 payload paths through one exclusive,
|
||||
digest-bound filesystem transaction and writes payload/terminal receipts
|
||||
below `/backup/status`.
|
||||
- The deployed common library, restore drill, offsite gate, and Host188 archive
|
||||
verifier gain the shared-lock behavior described above. Apply does not invoke
|
||||
their backup, drill, archive, or sync behavior.
|
||||
- Apply does not stop/start Gitea, create a backup container, call Google Drive,
|
||||
prune snapshots, restart a VM or service, change a database/network/firewall,
|
||||
or read/change a credential or secret.
|
||||
|
||||
## Reproducible Validation
|
||||
|
||||
Expanded focused and compatibility suite:
|
||||
|
||||
```sh
|
||||
pytest -q \
|
||||
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
|
||||
scripts/backup/tests/test_verify_offsite_full_sync_runtime.py \
|
||||
scripts/backup/tests/test_verify_host188_products_archive.py \
|
||||
scripts/backup/tests/test_backup_host188_products_contract.py \
|
||||
scripts/ops/tests/test_agent99_signoz_metadata_executor_runtime_entrypoint.py \
|
||||
scripts/ops/tests/test_db_bounded_executor_runtime_entrypoint.py \
|
||||
scripts/reboot-recovery/tests/test_agent99_live_preflight_decision.py \
|
||||
scripts/reboot-recovery/tests/test_agent99_remote_atomic_deploy_transport_contract.py \
|
||||
scripts/reboot-recovery/tests/test_host112_manager_recovery_contract.py
|
||||
```
|
||||
|
||||
Current result: `120 passed, 4 skipped`.
|
||||
|
||||
The shared-lock subset includes real `fcntl` contention for absolute,
|
||||
relative, symlink, forged-environment, inherited-exact-FD, inherited-wrong-FD,
|
||||
and Host188 archive paths. Current result: `18 passed, 31 deselected`.
|
||||
|
||||
Static, syntax, and repository checks:
|
||||
|
||||
```sh
|
||||
ruff check \
|
||||
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
|
||||
scripts/backup/verify-host188-products-archive.py
|
||||
bash -n \
|
||||
scripts/backup/common.sh \
|
||||
scripts/backup/gitea-full-backup-restore-drill.sh \
|
||||
scripts/backup/backup-offsite-readiness-gate.sh \
|
||||
scripts/backup/host110-backup-runtime-executor.sh
|
||||
git diff --check
|
||||
PATH=/Users/ogt/.pyenv/shims:$PATH bash scripts/ops/ansible-validate.sh
|
||||
```
|
||||
|
||||
`ansible-validate.sh` parses
|
||||
`ops/reboot-recovery/full-stack-backup-baseline.yml` as YAML metadata. It runs
|
||||
`ansible-playbook --syntax-check` only for actual playbooks. Current checks are
|
||||
green; `ansible-lint` is not installed and is reported as skipped.
|
||||
|
||||
Windows99 no-write PowerShell validation consists of:
|
||||
|
||||
1. parsing the complete candidate broker with Windows PowerShell;
|
||||
2. loading the candidate broker's actual function AST in the contract replay;
|
||||
3. replaying valid success, committed non-success, existing evidence rejection,
|
||||
terminal-missing reconciliation, and committed-unknown cases.
|
||||
|
||||
Current readback is parser `errorCount=0`, plus 38 existing-evidence
|
||||
rejections, 38 missing-field rejections, 6 type rejections,
|
||||
`terminalMissingReconciled=true`, `committedUnknownPreserved=true`,
|
||||
`remoteWritePerformed=false`, and `secretValuesRead=false`.
|
||||
|
||||
## Exact Identity And Diff Serialization
|
||||
|
||||
The central-review intake must bind all of the following independently:
|
||||
|
||||
- live Gitea feature ref `codex/host110-backup-runtime-v11-20260718`;
|
||||
- exact candidate Git SHA;
|
||||
- this tracked artifact path and exact artifact-bytes SHA-256;
|
||||
- exact binary diff SHA-256, path count, and insertion/deletion counts;
|
||||
- the 19/18/21 count contracts;
|
||||
- `scripts/backup/backup-gitea.sh` zero diff;
|
||||
- the Windows99 no-write replay result.
|
||||
|
||||
Exact diff bytes are serialized with:
|
||||
|
||||
```sh
|
||||
git diff --binary --full-index \
|
||||
e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3..<candidate_git_sha> |
|
||||
shasum -a 256
|
||||
```
|
||||
|
||||
No alternative diff serialization or untracked artifact bytes may be used for
|
||||
approval identity.
|
||||
|
||||
## Rollback
|
||||
|
||||
- Source rollback reverts the exact V11 diff to
|
||||
`e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3` without force-pushing.
|
||||
- Agent99 rollback restores prior runtime files and manifest and verifies prior
|
||||
hashes independently.
|
||||
- Before payload commit, Host110 rollback restores prior payload content,
|
||||
SHA-256, uid, gid, and mode under the exclusive lock and proves no run-owned
|
||||
destination temporary remains.
|
||||
- After payload commit, transport loss cannot trigger a blind Apply retry.
|
||||
Exact receipts and independent destination readback determine whether the
|
||||
state is committed or unknown; every uncertain state remains non-success.
|
||||
- Source rollback restores prior lock behavior. It does not execute a backup,
|
||||
restore drill, offsite sync, container lifecycle, or service restart.
|
||||
- Any identity, schema, type, receipt, reconciliation, lock, cleanup, or
|
||||
residue mismatch fails closed and cannot produce broker `verified`.
|
||||
@@ -9,14 +9,74 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
if [[ "${BASH_SOURCE[0]}" == /backup/scripts/* ]] && [ "${BACKUP_RUNTIME_SHARED_LOCK_HELD:-0}" != "1" ]; then
|
||||
resolve_backup_runtime_script_path() {
|
||||
local raw_path="$1" candidate="" resolved="" resolved_directory=""
|
||||
[ -n "${raw_path}" ] || return 1
|
||||
case "${raw_path}" in /*) candidate="${raw_path}" ;; *) candidate="${PWD}/${raw_path}" ;; esac
|
||||
if command -v realpath >/dev/null 2>&1; then
|
||||
resolved="$(realpath -e -- "${candidate}" 2>/dev/null || realpath -- "${candidate}" 2>/dev/null)" || resolved=""
|
||||
if [ -n "${resolved}" ]; then printf '%s\n' "${resolved}"; return 0; fi
|
||||
fi
|
||||
if command -v readlink >/dev/null 2>&1; then
|
||||
resolved="$(readlink -f -- "${candidate}" 2>/dev/null)" || resolved=""
|
||||
if [ -n "${resolved}" ]; then printf '%s\n' "${resolved}"; return 0; fi
|
||||
fi
|
||||
[ ! -L "${candidate}" ] || return 1
|
||||
resolved_directory="$(cd -P -- "$(dirname -- "${candidate}")" 2>/dev/null && pwd -P)" || return 1
|
||||
[ -e "${resolved_directory}/$(basename -- "${candidate}")" ] || return 1
|
||||
printf '%s/%s\n' "${resolved_directory}" "$(basename -- "${candidate}")"
|
||||
}
|
||||
|
||||
backup_runtime_inherited_fd_status() {
|
||||
local expected_path="$1" fd_path="" fd_target=""
|
||||
if [ -d "/proc/$$/fd" ]; then
|
||||
fd_path="/proc/$$/fd/197"
|
||||
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
|
||||
fd_target="$(resolve_backup_runtime_script_path "${fd_path}")" || return 2
|
||||
[ "${fd_target}" = "${expected_path}" ] || return 2
|
||||
return 0
|
||||
fi
|
||||
fd_path="/dev/fd/197"
|
||||
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
|
||||
command -v python3 >/dev/null 2>&1 || return 2
|
||||
fd_target="$(python3 - <<'PY'
|
||||
import fcntl
|
||||
import os
|
||||
|
||||
try:
|
||||
target = os.readlink("/dev/fd/197")
|
||||
except OSError:
|
||||
value = fcntl.fcntl(197, getattr(fcntl, "F_GETPATH", 50), b"\0" * 1024)
|
||||
target = os.fsdecode(value.split(b"\0", 1)[0])
|
||||
print(os.path.realpath(target))
|
||||
PY
|
||||
)" || return 2
|
||||
[ "${fd_target}" = "${expected_path}" ] || return 2
|
||||
}
|
||||
|
||||
runtime_script_path="$(resolve_backup_runtime_script_path "${BASH_SOURCE[0]:-}")" || {
|
||||
echo "backup runtime gate unavailable: script identity unresolved" >&2
|
||||
exit 69
|
||||
}
|
||||
if [[ "${runtime_script_path}" == /backup/scripts/* ]]; then
|
||||
command -v flock >/dev/null 2>&1 || { echo "backup runtime gate unavailable" >&2; exit 69; }
|
||||
[ ! -L /tmp/agent99-host110-backup-runtime.lock ] || { echo "backup runtime gate unsafe" >&2; exit 69; }
|
||||
(umask 077; : >> /tmp/agent99-host110-backup-runtime.lock)
|
||||
[ -f /tmp/agent99-host110-backup-runtime.lock ] && [ -O /tmp/agent99-host110-backup-runtime.lock ] || { echo "backup runtime gate owner invalid" >&2; exit 69; }
|
||||
exec 197>>/tmp/agent99-host110-backup-runtime.lock
|
||||
runtime_lock_path="$(resolve_backup_runtime_script_path /tmp/agent99-host110-backup-runtime.lock)" \
|
||||
|| { echo "backup runtime gate unavailable: lock identity unresolved" >&2; exit 69; }
|
||||
if backup_runtime_inherited_fd_status "${runtime_lock_path}"; then
|
||||
:
|
||||
else
|
||||
inherited_fd_status=$?
|
||||
if [ "${inherited_fd_status}" -eq 1 ]; then
|
||||
exec 197>>/tmp/agent99-host110-backup-runtime.lock
|
||||
else
|
||||
echo "backup runtime gate unsafe: inherited fd 197 identity mismatch" >&2
|
||||
exit 69
|
||||
fi
|
||||
fi
|
||||
flock -s -w 60 197 || { echo "backup runtime deployment is active; retry later" >&2; exit 75; }
|
||||
export BACKUP_RUNTIME_SHARED_LOCK_HELD=1
|
||||
fi
|
||||
|
||||
BACKUP_BASE="${BACKUP_BASE:-/backup}"
|
||||
|
||||
@@ -3,14 +3,78 @@
|
||||
|
||||
# Production backup entrypoints share this stable gate. Agent99 takes the
|
||||
# exclusive side while promoting a complete, digest-bound runtime bundle.
|
||||
resolve_backup_runtime_source_path() {
|
||||
local raw_path="$1"
|
||||
local candidate=""
|
||||
local resolved=""
|
||||
local resolved_directory=""
|
||||
[ -n "${raw_path}" ] || return 1
|
||||
case "${raw_path}" in
|
||||
/*) candidate="${raw_path}" ;;
|
||||
*) candidate="${PWD}/${raw_path}" ;;
|
||||
esac
|
||||
if command -v realpath >/dev/null 2>&1; then
|
||||
resolved="$(realpath -e -- "${candidate}" 2>/dev/null || realpath -- "${candidate}" 2>/dev/null)" || resolved=""
|
||||
if [ -n "${resolved}" ]; then
|
||||
printf '%s\n' "${resolved}"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
if command -v readlink >/dev/null 2>&1; then
|
||||
resolved="$(readlink -f -- "${candidate}" 2>/dev/null)" || resolved=""
|
||||
if [ -n "${resolved}" ]; then
|
||||
printf '%s\n' "${resolved}"
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
[ ! -L "${candidate}" ] || return 1
|
||||
resolved_directory="$(cd -P -- "$(dirname -- "${candidate}")" 2>/dev/null && pwd -P)" || return 1
|
||||
[ -e "${resolved_directory}/$(basename -- "${candidate}")" ] || return 1
|
||||
printf '%s/%s\n' "${resolved_directory}" "$(basename -- "${candidate}")"
|
||||
}
|
||||
|
||||
backup_runtime_inherited_fd_status() {
|
||||
local expected_path="$1"
|
||||
local fd_path=""
|
||||
local fd_target=""
|
||||
if [ -d "/proc/$$/fd" ]; then
|
||||
fd_path="/proc/$$/fd/197"
|
||||
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
|
||||
fd_target="$(resolve_backup_runtime_source_path "${fd_path}")" || return 2
|
||||
[ "${fd_target}" = "${expected_path}" ] || return 2
|
||||
return 0
|
||||
fi
|
||||
fd_path="/dev/fd/197"
|
||||
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
|
||||
command -v python3 >/dev/null 2>&1 || return 2
|
||||
fd_target="$(python3 - <<'PY'
|
||||
import fcntl
|
||||
import os
|
||||
|
||||
try:
|
||||
target = os.readlink("/dev/fd/197")
|
||||
except OSError:
|
||||
value = fcntl.fcntl(197, getattr(fcntl, "F_GETPATH", 50), b"\0" * 1024)
|
||||
target = os.fsdecode(value.split(b"\0", 1)[0])
|
||||
print(os.path.realpath(target))
|
||||
PY
|
||||
)" || return 2
|
||||
[ "${fd_target}" = "${expected_path}" ] || return 2
|
||||
}
|
||||
|
||||
acquire_backup_runtime_shared_lock() {
|
||||
local caller_path="${BASH_SOURCE[1]:-${BASH_SOURCE[0]}}"
|
||||
local caller_path=""
|
||||
local lock_path="/tmp/agent99-host110-backup-runtime.lock"
|
||||
local lock_canonical=""
|
||||
local inherited_fd_status=0
|
||||
caller_path="$(resolve_backup_runtime_source_path "${BASH_SOURCE[1]:-${BASH_SOURCE[0]:-}}")" || {
|
||||
echo "backup runtime gate unavailable: caller identity unresolved" >&2
|
||||
return 69
|
||||
}
|
||||
case "${caller_path}" in
|
||||
/backup/scripts/*) ;;
|
||||
*) return 0 ;;
|
||||
esac
|
||||
[ "${BACKUP_RUNTIME_SHARED_LOCK_HELD:-0}" != "1" ] || return 0
|
||||
command -v flock >/dev/null 2>&1 || {
|
||||
echo "backup runtime gate unavailable: flock missing" >&2
|
||||
return 69
|
||||
@@ -21,12 +85,25 @@ acquire_backup_runtime_shared_lock() {
|
||||
}
|
||||
(umask 077; : >> "${lock_path}") || return 69
|
||||
[ -f "${lock_path}" ] && [ -O "${lock_path}" ] || return 69
|
||||
exec 197>>"${lock_path}"
|
||||
lock_canonical="$(resolve_backup_runtime_source_path "${lock_path}")" || {
|
||||
echo "backup runtime gate unavailable: lock identity unresolved" >&2
|
||||
return 69
|
||||
}
|
||||
if backup_runtime_inherited_fd_status "${lock_canonical}"; then
|
||||
:
|
||||
else
|
||||
inherited_fd_status=$?
|
||||
if [ "${inherited_fd_status}" -eq 1 ]; then
|
||||
exec 197>>"${lock_path}"
|
||||
else
|
||||
echo "backup runtime gate unsafe: inherited fd 197 identity mismatch" >&2
|
||||
return 69
|
||||
fi
|
||||
fi
|
||||
flock -s -w 60 197 || {
|
||||
echo "backup runtime deployment is active; retry later" >&2
|
||||
return 75
|
||||
}
|
||||
export BACKUP_RUNTIME_SHARED_LOCK_HELD=1
|
||||
}
|
||||
|
||||
acquire_backup_runtime_shared_lock || {
|
||||
|
||||
@@ -5,14 +5,74 @@
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
if [[ "${BASH_SOURCE[0]}" == /backup/scripts/* ]] && [ "${BACKUP_RUNTIME_SHARED_LOCK_HELD:-0}" != "1" ]; then
|
||||
resolve_backup_runtime_script_path() {
|
||||
local raw_path="$1" candidate="" resolved="" resolved_directory=""
|
||||
[ -n "${raw_path}" ] || return 1
|
||||
case "${raw_path}" in /*) candidate="${raw_path}" ;; *) candidate="${PWD}/${raw_path}" ;; esac
|
||||
if command -v realpath >/dev/null 2>&1; then
|
||||
resolved="$(realpath -e -- "${candidate}" 2>/dev/null || realpath -- "${candidate}" 2>/dev/null)" || resolved=""
|
||||
if [ -n "${resolved}" ]; then printf '%s\n' "${resolved}"; return 0; fi
|
||||
fi
|
||||
if command -v readlink >/dev/null 2>&1; then
|
||||
resolved="$(readlink -f -- "${candidate}" 2>/dev/null)" || resolved=""
|
||||
if [ -n "${resolved}" ]; then printf '%s\n' "${resolved}"; return 0; fi
|
||||
fi
|
||||
[ ! -L "${candidate}" ] || return 1
|
||||
resolved_directory="$(cd -P -- "$(dirname -- "${candidate}")" 2>/dev/null && pwd -P)" || return 1
|
||||
[ -e "${resolved_directory}/$(basename -- "${candidate}")" ] || return 1
|
||||
printf '%s/%s\n' "${resolved_directory}" "$(basename -- "${candidate}")"
|
||||
}
|
||||
|
||||
backup_runtime_inherited_fd_status() {
|
||||
local expected_path="$1" fd_path="" fd_target=""
|
||||
if [ -d "/proc/$$/fd" ]; then
|
||||
fd_path="/proc/$$/fd/197"
|
||||
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
|
||||
fd_target="$(resolve_backup_runtime_script_path "${fd_path}")" || return 2
|
||||
[ "${fd_target}" = "${expected_path}" ] || return 2
|
||||
return 0
|
||||
fi
|
||||
fd_path="/dev/fd/197"
|
||||
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
|
||||
command -v python3 >/dev/null 2>&1 || return 2
|
||||
fd_target="$(python3 - <<'PY'
|
||||
import fcntl
|
||||
import os
|
||||
|
||||
try:
|
||||
target = os.readlink("/dev/fd/197")
|
||||
except OSError:
|
||||
value = fcntl.fcntl(197, getattr(fcntl, "F_GETPATH", 50), b"\0" * 1024)
|
||||
target = os.fsdecode(value.split(b"\0", 1)[0])
|
||||
print(os.path.realpath(target))
|
||||
PY
|
||||
)" || return 2
|
||||
[ "${fd_target}" = "${expected_path}" ] || return 2
|
||||
}
|
||||
|
||||
runtime_script_path="$(resolve_backup_runtime_script_path "${BASH_SOURCE[0]:-}")" || {
|
||||
echo "backup runtime gate unavailable: script identity unresolved" >&2
|
||||
exit 69
|
||||
}
|
||||
if [[ "${runtime_script_path}" == /backup/scripts/* ]]; then
|
||||
command -v flock >/dev/null 2>&1 || { echo "backup runtime gate unavailable" >&2; exit 69; }
|
||||
[ ! -L /tmp/agent99-host110-backup-runtime.lock ] || { echo "backup runtime gate unsafe" >&2; exit 69; }
|
||||
(umask 077; : >> /tmp/agent99-host110-backup-runtime.lock)
|
||||
[ -f /tmp/agent99-host110-backup-runtime.lock ] && [ -O /tmp/agent99-host110-backup-runtime.lock ] || { echo "backup runtime gate owner invalid" >&2; exit 69; }
|
||||
exec 197>>/tmp/agent99-host110-backup-runtime.lock
|
||||
runtime_lock_path="$(resolve_backup_runtime_script_path /tmp/agent99-host110-backup-runtime.lock)" \
|
||||
|| { echo "backup runtime gate unavailable: lock identity unresolved" >&2; exit 69; }
|
||||
if backup_runtime_inherited_fd_status "${runtime_lock_path}"; then
|
||||
:
|
||||
else
|
||||
inherited_fd_status=$?
|
||||
if [ "${inherited_fd_status}" -eq 1 ]; then
|
||||
exec 197>>/tmp/agent99-host110-backup-runtime.lock
|
||||
else
|
||||
echo "backup runtime gate unsafe: inherited fd 197 identity mismatch" >&2
|
||||
exit 69
|
||||
fi
|
||||
fi
|
||||
flock -s -w 60 197 || { echo "backup runtime deployment is active; retry later" >&2; exit 75; }
|
||||
export BACKUP_RUNTIME_SHARED_LOCK_HELD=1
|
||||
fi
|
||||
|
||||
DUMP_ZIP="${AIOPS_GITEA_FULL_BACKUP_DRILL_DUMP_ZIP:-}"
|
||||
|
||||
@@ -45,8 +45,12 @@ VERIFIER_DIGEST=""
|
||||
STAGE_DIR=""
|
||||
ROLLBACK_DIR=""
|
||||
RECEIPT_PATH=""
|
||||
TERMINAL_RECEIPT_PATH=""
|
||||
APPLY_STARTED=0
|
||||
APPLY_COMPLETE=0
|
||||
COMMIT_CRITICAL=0
|
||||
DEFERRED_SIGNAL=""
|
||||
DEFERRED_SIGNAL_EXIT=0
|
||||
ROLLBACK_ATTEMPTED=0
|
||||
ROLLBACK_PERFORMED=0
|
||||
ROLLBACK_VERIFIED=0
|
||||
@@ -103,10 +107,17 @@ case "$MODE" in check|apply|verify) ;; *) fail "invalid_mode" ;; esac
|
||||
expected_source_stage="/tmp/agent99-host110-backup-runtime-${RUN_ID}"
|
||||
[ "$SOURCE_STAGE" = "$expected_source_stage" ] || fail "invalid_source_stage"
|
||||
RECEIPT_PATH="$STATUS_ROOT/agent99-backup-runtime-deploy-${RUN_ID}.json"
|
||||
TERMINAL_RECEIPT_PATH="$STATUS_ROOT/agent99-backup-runtime-terminal-${RUN_ID}.json"
|
||||
if [ "${HOST110_BACKUP_RUNTIME_ENABLE_TEST_FAULTS:-}" = "1" ] \
|
||||
&& [ "$DEST_ROOT" != "/backup/scripts" ]; then
|
||||
FAULT_INJECTION_ENABLED=1
|
||||
fi
|
||||
if [ "$FAULT_INJECTION_ENABLED" -eq 1 ]; then
|
||||
case "${HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL:-}" in ""|INT|HUP|TERM) ;; *) fail "invalid_test_commit_signal" ;; esac
|
||||
case "${HOST110_BACKUP_RUNTIME_TEST_TERMINAL_SIGNAL:-}" in ""|INT|HUP|TERM) ;; *) fail "invalid_test_terminal_signal" ;; esac
|
||||
case "${HOST110_BACKUP_RUNTIME_TEST_TERMINAL_RECEIPT_FAULT:-}" in ""|write|link|fsync|readback) ;; *) fail "invalid_test_terminal_receipt_fault" ;; esac
|
||||
case "${HOST110_BACKUP_RUNTIME_TEST_CLEANUP_FAULT:-}" in ""|stage|rollback_prestate) ;; *) fail "invalid_test_cleanup_fault" ;; esac
|
||||
fi
|
||||
|
||||
for command_name in awk bash cp dirname flock grep hostname id install mv pgrep python3 readlink rm sha256sum stat timeout tr; do
|
||||
command -v "$command_name" >/dev/null 2>&1 || fail "required_command_missing_${command_name}"
|
||||
@@ -314,9 +325,9 @@ rollback_attempted = sys.argv[6] == "1"
|
||||
rollback_performed = sys.argv[7] == "1"
|
||||
rollback_verified = sys.argv[8] == "1"
|
||||
zero_residue_verified = sys.argv[9] == "1"
|
||||
if status not in {"verified", "failed_rolled_back", "rollback_unverified"}:
|
||||
if status not in {"payload_verified", "failed_rolled_back", "rollback_unverified"}:
|
||||
raise SystemExit(78)
|
||||
if status == "verified" and (rollback_attempted or rollback_performed or rollback_verified or not zero_residue_verified):
|
||||
if status == "payload_verified" and (rollback_attempted or rollback_performed or rollback_verified or not zero_residue_verified):
|
||||
raise SystemExit(78)
|
||||
if status == "failed_rolled_back" and not (
|
||||
rollback_attempted and rollback_performed and rollback_verified and zero_residue_verified
|
||||
@@ -325,7 +336,7 @@ if status == "failed_rolled_back" and not (
|
||||
if status == "rollback_unverified" and (not rollback_attempted or rollback_verified):
|
||||
raise SystemExit(78)
|
||||
document = {
|
||||
"schemaVersion": "agent99_host110_backup_runtime_deploy_receipt_v2",
|
||||
"schemaVersion": "agent99_host110_backup_runtime_deploy_receipt_v3",
|
||||
"status": status,
|
||||
"sourceRevision": sys.argv[3],
|
||||
"sourceHead": sys.argv[4],
|
||||
@@ -337,6 +348,7 @@ document = {
|
||||
"rollbackPerformed": rollback_performed,
|
||||
"rollbackVerified": rollback_verified,
|
||||
"zeroResidueVerified": zero_residue_verified,
|
||||
"zeroResidueScope": "run_owned_destination_temps",
|
||||
"verifierSha256": sys.argv[10],
|
||||
"executorHost": "192.168.0.110",
|
||||
"productionServiceRestarted": False,
|
||||
@@ -349,7 +361,7 @@ temporary = path.with_name(f".{path.name}.tmp-{os.getpid()}")
|
||||
fault_enabled = sys.argv[11] == "1"
|
||||
fault = (
|
||||
os.environ.get("HOST110_BACKUP_RUNTIME_TEST_RECEIPT_FAULT", "")
|
||||
if status == "verified" and fault_enabled
|
||||
if status == "payload_verified" and fault_enabled
|
||||
else ""
|
||||
)
|
||||
if fault not in {"", "write", "link", "fsync", "readback"}:
|
||||
@@ -412,6 +424,172 @@ PY
|
||||
return 0
|
||||
}
|
||||
|
||||
write_terminal_receipt() {
|
||||
local terminal_status="$1"
|
||||
local stage_cleanup_verified="$2"
|
||||
local rollback_prestate_cleanup_verified="$3"
|
||||
local deferred_signal="$4"
|
||||
local terminal_exit_code="$5"
|
||||
install -d -m 700 "$STATUS_ROOT" || return 1
|
||||
[ -d "$STATUS_ROOT" ] && [ ! -L "$STATUS_ROOT" ] || return 1
|
||||
[ "$(stat -c '%U:%G:%a' "$STATUS_ROOT")" = "wooo:wooo:700" ] || return 1
|
||||
if ! python3 - "$TERMINAL_RECEIPT_PATH" "$RECEIPT_PATH" "$terminal_status" \
|
||||
"$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID" "$stage_cleanup_verified" \
|
||||
"$rollback_prestate_cleanup_verified" "$deferred_signal" "$terminal_exit_code" \
|
||||
"$VERIFIER_DIGEST" "$FAULT_INJECTION_ENABLED" <<'PY'
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
path = Path(sys.argv[1])
|
||||
payload_receipt_path = Path(sys.argv[2])
|
||||
status = sys.argv[3]
|
||||
source_revision = sys.argv[4]
|
||||
source_head = sys.argv[5]
|
||||
run_id = sys.argv[6]
|
||||
stage_cleanup_verified = sys.argv[7] == "1"
|
||||
rollback_prestate_cleanup_verified = sys.argv[8] == "1"
|
||||
deferred_signal = sys.argv[9]
|
||||
terminal_exit_code = int(sys.argv[10])
|
||||
verifier_digest = sys.argv[11]
|
||||
fault_enabled = sys.argv[12] == "1"
|
||||
signal_exit_codes = {"INT": 130, "HUP": 129, "TERM": 143}
|
||||
|
||||
terminal_signal = os.environ.get("HOST110_BACKUP_RUNTIME_TEST_TERMINAL_SIGNAL", "") if fault_enabled else ""
|
||||
terminal_fault = os.environ.get("HOST110_BACKUP_RUNTIME_TEST_TERMINAL_RECEIPT_FAULT", "") if fault_enabled else ""
|
||||
if terminal_signal:
|
||||
os.kill(os.getppid(), getattr(signal, f"SIG{terminal_signal}"))
|
||||
time.sleep(0.05)
|
||||
|
||||
try:
|
||||
payload_receipt = json.loads(payload_receipt_path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
raise SystemExit(79)
|
||||
if not (
|
||||
payload_receipt.get("schemaVersion") == "agent99_host110_backup_runtime_deploy_receipt_v3"
|
||||
and payload_receipt.get("status") == "payload_verified"
|
||||
and payload_receipt.get("sourceRevision") == source_revision
|
||||
and payload_receipt.get("sourceHead") == source_head
|
||||
and payload_receipt.get("runId") == run_id
|
||||
and payload_receipt.get("zeroResidueVerified") is True
|
||||
):
|
||||
raise SystemExit(79)
|
||||
|
||||
if status == "verified":
|
||||
valid = (
|
||||
stage_cleanup_verified
|
||||
and rollback_prestate_cleanup_verified
|
||||
and not deferred_signal
|
||||
and terminal_exit_code == 0
|
||||
)
|
||||
elif status == "cleanup_pending":
|
||||
valid = (
|
||||
not (stage_cleanup_verified and rollback_prestate_cleanup_verified)
|
||||
and deferred_signal in {"", *signal_exit_codes}
|
||||
and terminal_exit_code != 0
|
||||
)
|
||||
elif status == "committed_signal_deferred":
|
||||
valid = (
|
||||
stage_cleanup_verified
|
||||
and rollback_prestate_cleanup_verified
|
||||
and deferred_signal in signal_exit_codes
|
||||
and terminal_exit_code == signal_exit_codes[deferred_signal]
|
||||
)
|
||||
else:
|
||||
valid = False
|
||||
if not valid:
|
||||
raise SystemExit(79)
|
||||
|
||||
document = {
|
||||
"schemaVersion": "agent99_host110_backup_runtime_terminal_receipt_v1",
|
||||
"status": status,
|
||||
"ok": status == "verified",
|
||||
"payloadCommitted": True,
|
||||
"payloadReceipt": str(payload_receipt_path),
|
||||
"sourceRevision": source_revision,
|
||||
"sourceHead": source_head,
|
||||
"runId": run_id,
|
||||
"fileCount": 18,
|
||||
"backupScriptFileCount": 17,
|
||||
"backupHealthExporterIncluded": True,
|
||||
"stageCleanupVerified": stage_cleanup_verified,
|
||||
"rollbackPrestateCleanupVerified": rollback_prestate_cleanup_verified,
|
||||
"cleanupVerified": stage_cleanup_verified and rollback_prestate_cleanup_verified,
|
||||
"independentVerifierVerified": True,
|
||||
"zeroResidueScope": "run_owned_destination_temps_and_internal_stage_prestate",
|
||||
"deferredSignal": deferred_signal or None,
|
||||
"terminalExitCode": terminal_exit_code,
|
||||
"verifierSha256": verifier_digest,
|
||||
"executorHost": "192.168.0.110",
|
||||
"productionServiceRestarted": False,
|
||||
"secretValuesRead": False,
|
||||
"writtenAt": int(time.time()),
|
||||
}
|
||||
payload = (json.dumps(document, ensure_ascii=True, sort_keys=True, separators=(",", ":")) + "\n").encode("utf-8")
|
||||
expected_digest = hashlib.sha256(payload).hexdigest()
|
||||
temporary = path.with_name(f".{path.name}.tmp-{os.getpid()}")
|
||||
created_final = False
|
||||
directory_fd = None
|
||||
try:
|
||||
if os.path.lexists(path) or os.path.lexists(temporary):
|
||||
raise FileExistsError(path)
|
||||
if terminal_fault == "write":
|
||||
raise OSError("fault_injected_terminal_receipt_write")
|
||||
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
with os.fdopen(descriptor, "wb") as handle:
|
||||
if handle.write(payload) != len(payload):
|
||||
raise OSError("terminal_receipt_short_write")
|
||||
handle.flush()
|
||||
if terminal_fault == "fsync":
|
||||
raise OSError("fault_injected_terminal_receipt_fsync")
|
||||
os.fsync(handle.fileno())
|
||||
os.chmod(temporary, 0o600)
|
||||
temporary_readback = temporary.read_bytes()
|
||||
if temporary_readback != payload or hashlib.sha256(temporary_readback).hexdigest() != expected_digest:
|
||||
raise OSError("terminal_receipt_temporary_readback_failed")
|
||||
if terminal_fault == "link":
|
||||
raise OSError("fault_injected_terminal_receipt_link")
|
||||
os.link(temporary, path)
|
||||
created_final = True
|
||||
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
|
||||
directory_fd = os.open(path.parent, directory_flags)
|
||||
os.fsync(directory_fd)
|
||||
if terminal_fault == "readback":
|
||||
raise OSError("fault_injected_terminal_receipt_readback")
|
||||
readback = path.read_bytes()
|
||||
if readback != payload or hashlib.sha256(readback).hexdigest() != expected_digest:
|
||||
raise OSError("terminal_receipt_final_readback_failed")
|
||||
if json.loads(readback.decode("utf-8")) != document:
|
||||
raise OSError("terminal_receipt_document_readback_failed")
|
||||
temporary.unlink()
|
||||
os.fsync(directory_fd)
|
||||
except BaseException:
|
||||
try:
|
||||
if created_final:
|
||||
path.unlink(missing_ok=True)
|
||||
temporary.unlink(missing_ok=True)
|
||||
if directory_fd is None and path.parent.is_dir():
|
||||
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
|
||||
directory_fd = os.open(path.parent, directory_flags)
|
||||
if directory_fd is not None:
|
||||
os.fsync(directory_fd)
|
||||
finally:
|
||||
raise
|
||||
finally:
|
||||
if directory_fd is not None:
|
||||
os.close(directory_fd)
|
||||
PY
|
||||
then
|
||||
return 1
|
||||
fi
|
||||
[ -f "$TERMINAL_RECEIPT_PATH" ] && [ ! -L "$TERMINAL_RECEIPT_PATH" ] || return 1
|
||||
return 0
|
||||
}
|
||||
|
||||
record_prestate() {
|
||||
local name dest_path digest metadata
|
||||
: > "$ROLLBACK_DIR/prestate.tsv" || return 74
|
||||
@@ -530,6 +708,35 @@ rollback_transaction() {
|
||||
return 1
|
||||
}
|
||||
|
||||
handle_agent99_signal() {
|
||||
local signal_name="$1"
|
||||
local signal_exit_code="$2"
|
||||
if [ "$COMMIT_CRITICAL" -eq 1 ] || [ "$APPLY_COMPLETE" -eq 1 ]; then
|
||||
if [ -z "$DEFERRED_SIGNAL" ]; then
|
||||
DEFERRED_SIGNAL="$signal_name"
|
||||
DEFERRED_SIGNAL_EXIT="$signal_exit_code"
|
||||
fi
|
||||
return 0
|
||||
fi
|
||||
exit "$signal_exit_code"
|
||||
}
|
||||
|
||||
cleanup_postcommit_directory() {
|
||||
local cleanup_kind="$1"
|
||||
local cleanup_path="$2"
|
||||
local fault=""
|
||||
if [ "$FAULT_INJECTION_ENABLED" -eq 1 ]; then
|
||||
fault="${HOST110_BACKUP_RUNTIME_TEST_CLEANUP_FAULT:-}"
|
||||
fi
|
||||
if [ "$fault" = "$cleanup_kind" ]; then
|
||||
return 1
|
||||
fi
|
||||
rm -rf -- "$cleanup_path" || return 1
|
||||
[ ! -e "$cleanup_path" ] && [ ! -L "$cleanup_path" ] || return 1
|
||||
fsync_paths_and_parents "$cleanup_path" || return 1
|
||||
[ ! -e "$cleanup_path" ] && [ ! -L "$cleanup_path" ]
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
local exit_status=$?
|
||||
local rollback_status="rollback_unverified"
|
||||
@@ -544,10 +751,12 @@ cleanup() {
|
||||
failure_receipt_written=1
|
||||
fi
|
||||
fi
|
||||
if [ -n "$STAGE_DIR" ]; then
|
||||
if [ "$APPLY_COMPLETE" -ne 1 ] && [ -n "$STAGE_DIR" ]; then
|
||||
rm -rf -- "$STAGE_DIR" || true
|
||||
fi
|
||||
if [ "$ROLLBACK_VERIFIED" -eq 1 ] && [ "$failure_receipt_written" -eq 1 ]; then
|
||||
if [ "$APPLY_COMPLETE" -ne 1 ] \
|
||||
&& [ "$ROLLBACK_VERIFIED" -eq 1 ] \
|
||||
&& [ "$failure_receipt_written" -eq 1 ]; then
|
||||
if [ -n "$ROLLBACK_DIR" ]; then
|
||||
rm -rf -- "$ROLLBACK_DIR" || true
|
||||
fi
|
||||
@@ -558,17 +767,19 @@ cleanup() {
|
||||
validate_source || fail "source_validation_failed"
|
||||
|
||||
if [ "$MODE" = "check" ]; then
|
||||
printf '{"schemaVersion":"agent99_host110_backup_runtime_executor_v1","mode":"check","ok":true,"sourceRevision":"%s","sourceHead":"%s","fileCount":18,"backupScriptFileCount":17,"backupHealthExporterIncluded":true,"remoteWritePerformed":false}\n' "$SOURCE_REVISION" "$SOURCE_HEAD"
|
||||
printf '{"schemaVersion":"agent99_host110_backup_runtime_executor_v1","mode":"check","ok":true,"sourceRevision":"%s","sourceHead":"%s","runId":"%s","fileCount":18,"backupScriptFileCount":17,"backupHealthExporterIncluded":true,"remoteWritePerformed":false}\n' "$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$MODE" = "verify" ]; then
|
||||
verify_destination || fail "destination_verification_failed"
|
||||
printf '{"schemaVersion":"agent99_host110_backup_runtime_executor_v1","mode":"verify","ok":true,"sourceRevision":"%s","sourceHead":"%s","fileCount":18,"backupScriptFileCount":17,"backupHealthExporterIncluded":true,"remoteWritePerformed":false}\n' "$SOURCE_REVISION" "$SOURCE_HEAD"
|
||||
printf '{"schemaVersion":"agent99_host110_backup_runtime_executor_v1","mode":"verify","ok":true,"sourceRevision":"%s","sourceHead":"%s","runId":"%s","fileCount":18,"backupScriptFileCount":17,"backupHealthExporterIncluded":true,"remoteWritePerformed":false}\n' "$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[ ! -e "$RECEIPT_PATH" ] && [ ! -L "$RECEIPT_PATH" ] || fail "run_identity_receipt_exists"
|
||||
[ ! -e "$TERMINAL_RECEIPT_PATH" ] && [ ! -L "$TERMINAL_RECEIPT_PATH" ] \
|
||||
|| fail "run_identity_terminal_receipt_exists"
|
||||
STAGE_DIR="$STAGE_ROOT/$RUN_ID"
|
||||
ROLLBACK_DIR="$ROLLBACK_ROOT/$RUN_ID"
|
||||
[ ! -e "$STAGE_DIR" ] || fail "run_identity_stage_exists"
|
||||
@@ -589,8 +800,9 @@ fi
|
||||
|
||||
install -d -m 700 "$STAGE_DIR" "$ROLLBACK_DIR"
|
||||
trap cleanup EXIT
|
||||
trap 'exit 130' INT
|
||||
trap 'exit 143' TERM HUP
|
||||
trap 'handle_agent99_signal INT 130' INT
|
||||
trap 'handle_agent99_signal HUP 129' HUP
|
||||
trap 'handle_agent99_signal TERM 143' TERM
|
||||
|
||||
for name in "${PAYLOAD_FILES[@]}"; do
|
||||
install -m 700 "$SOURCE_STAGE/$name" "$STAGE_DIR/$name"
|
||||
@@ -651,43 +863,116 @@ PY
|
||||
|
||||
verify_run_owned_temp_absence || fail "post_apply_temp_residue_detected"
|
||||
RUN_TEMP_RESIDUE_VERIFIED=1
|
||||
if ! write_receipt "verified" 0 0 0 "$RUN_TEMP_RESIDUE_VERIFIED"; then
|
||||
COMMIT_CRITICAL=1
|
||||
if ! write_receipt "payload_verified" 0 0 0 "$RUN_TEMP_RESIDUE_VERIFIED"; then
|
||||
COMMIT_CRITICAL=0
|
||||
fail "durable_receipt_failed"
|
||||
fi
|
||||
receipt_path="$RECEIPT_PATH"
|
||||
if [ "$FAULT_INJECTION_ENABLED" -eq 1 ] \
|
||||
&& [ -n "${HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL:-}" ]; then
|
||||
case "$HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL" in
|
||||
INT|HUP|TERM) kill -s "$HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL" "$$" ;;
|
||||
esac
|
||||
fi
|
||||
APPLY_COMPLETE=1
|
||||
trap - EXIT HUP INT TERM
|
||||
COMMIT_CRITICAL=0
|
||||
stage_cleanup_verified=0
|
||||
rollback_prestate_cleanup_verified=0
|
||||
if rm -rf -- "$STAGE_DIR" && [ ! -e "$STAGE_DIR" ] && [ ! -L "$STAGE_DIR" ]; then
|
||||
if cleanup_postcommit_directory "stage" "$STAGE_DIR"; then
|
||||
stage_cleanup_verified=1
|
||||
fi
|
||||
if rm -rf -- "$ROLLBACK_DIR" && [ ! -e "$ROLLBACK_DIR" ] && [ ! -L "$ROLLBACK_DIR" ]; then
|
||||
if cleanup_postcommit_directory "rollback_prestate" "$ROLLBACK_DIR"; then
|
||||
rollback_prestate_cleanup_verified=1
|
||||
fi
|
||||
STAGE_DIR=""
|
||||
ROLLBACK_DIR=""
|
||||
python3 - "$SOURCE_REVISION" "$SOURCE_HEAD" "$receipt_path" "$verifier_result" \
|
||||
"$stage_cleanup_verified" "$rollback_prestate_cleanup_verified" <<'PY'
|
||||
terminal_status="verified"
|
||||
terminal_exit_code=0
|
||||
# Cleanup has completed. Freeze terminal signal state before the immutable
|
||||
# terminal writer so its receipt, stdout, and exit status cannot diverge.
|
||||
trap '' HUP INT TERM
|
||||
if [ "$stage_cleanup_verified" -ne 1 ] || [ "$rollback_prestate_cleanup_verified" -ne 1 ]; then
|
||||
terminal_status="cleanup_pending"
|
||||
terminal_exit_code=75
|
||||
elif [ -n "$DEFERRED_SIGNAL" ]; then
|
||||
terminal_status="committed_signal_deferred"
|
||||
terminal_exit_code="$DEFERRED_SIGNAL_EXIT"
|
||||
fi
|
||||
if ! write_terminal_receipt "$terminal_status" "$stage_cleanup_verified" \
|
||||
"$rollback_prestate_cleanup_verified" "$DEFERRED_SIGNAL" "$terminal_exit_code"; then
|
||||
trap - EXIT
|
||||
python3 - "$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID" "$RECEIPT_PATH" \
|
||||
"$TERMINAL_RECEIPT_PATH" "$stage_cleanup_verified" \
|
||||
"$rollback_prestate_cleanup_verified" "$DEFERRED_SIGNAL" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
print(json.dumps({
|
||||
"schemaVersion": "agent99_host110_backup_runtime_executor_v1",
|
||||
"mode": "apply",
|
||||
"ok": True,
|
||||
"status": "terminal_receipt_failed",
|
||||
"ok": False,
|
||||
"sourceRevision": sys.argv[1],
|
||||
"sourceHead": sys.argv[2],
|
||||
"runId": sys.argv[3],
|
||||
"fileCount": 18,
|
||||
"backupScriptFileCount": 17,
|
||||
"backupHealthExporterIncluded": True,
|
||||
"payloadCommitted": True,
|
||||
"rollbackAttempted": False,
|
||||
"rollbackPerformed": False,
|
||||
"rollbackVerified": False,
|
||||
"zeroResidueVerified": True,
|
||||
"stageCleanupVerified": sys.argv[5] == "1",
|
||||
"rollbackPrestateCleanupVerified": sys.argv[6] == "1",
|
||||
"receipt": sys.argv[3],
|
||||
"verifier": json.loads(sys.argv[4]),
|
||||
"zeroResidueScope": "run_owned_destination_temps",
|
||||
"stageCleanupVerified": sys.argv[6] == "1",
|
||||
"rollbackPrestateCleanupVerified": sys.argv[7] == "1",
|
||||
"receipt": sys.argv[4],
|
||||
"terminalReceipt": sys.argv[5],
|
||||
"terminalReceiptPublished": False,
|
||||
"terminalExitCode": 76,
|
||||
"deferredSignal": sys.argv[8] or None,
|
||||
"independentVerifierVerified": True,
|
||||
"verifier": None,
|
||||
}, ensure_ascii=True, sort_keys=True))
|
||||
PY
|
||||
printf 'HOST110_BACKUP_RUNTIME_OK=0\nERROR=terminal_receipt_failed\n' >&2
|
||||
exit 76
|
||||
fi
|
||||
if [ "$stage_cleanup_verified" -eq 1 ]; then STAGE_DIR=""; fi
|
||||
if [ "$rollback_prestate_cleanup_verified" -eq 1 ]; then ROLLBACK_DIR=""; fi
|
||||
trap - EXIT
|
||||
python3 - "$SOURCE_REVISION" "$SOURCE_HEAD" "$RECEIPT_PATH" "$TERMINAL_RECEIPT_PATH" \
|
||||
"$verifier_result" "$stage_cleanup_verified" "$rollback_prestate_cleanup_verified" \
|
||||
"$terminal_status" "$terminal_exit_code" "$DEFERRED_SIGNAL" "$RUN_ID" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
|
||||
status = sys.argv[8]
|
||||
verifier = json.loads(sys.argv[5])
|
||||
print(json.dumps({
|
||||
"schemaVersion": "agent99_host110_backup_runtime_executor_v1",
|
||||
"mode": "apply",
|
||||
"status": status,
|
||||
"ok": status == "verified",
|
||||
"sourceRevision": sys.argv[1],
|
||||
"sourceHead": sys.argv[2],
|
||||
"runId": sys.argv[11],
|
||||
"fileCount": 18,
|
||||
"backupScriptFileCount": 17,
|
||||
"backupHealthExporterIncluded": True,
|
||||
"payloadCommitted": True,
|
||||
"rollbackAttempted": False,
|
||||
"rollbackPerformed": False,
|
||||
"rollbackVerified": False,
|
||||
"zeroResidueVerified": True,
|
||||
"zeroResidueScope": "run_owned_destination_temps",
|
||||
"stageCleanupVerified": sys.argv[6] == "1",
|
||||
"rollbackPrestateCleanupVerified": sys.argv[7] == "1",
|
||||
"receipt": sys.argv[3],
|
||||
"terminalReceipt": sys.argv[4],
|
||||
"terminalReceiptPublished": True,
|
||||
"terminalExitCode": int(sys.argv[9]),
|
||||
"deferredSignal": sys.argv[10] or None,
|
||||
"independentVerifierVerified": True,
|
||||
"verifier": verifier if status == "verified" else None,
|
||||
}, ensure_ascii=True, sort_keys=True))
|
||||
PY
|
||||
exit "$terminal_exit_code"
|
||||
|
||||
@@ -27,11 +27,46 @@ RUNTIME_LOCK = Path("/tmp/agent99-host110-backup-runtime.lock")
|
||||
|
||||
|
||||
def acquire_runtime_lock():
|
||||
if Path(__file__).resolve().parent != Path("/backup/scripts") or os.environ.get("BACKUP_RUNTIME_SHARED_LOCK_HELD") == "1":
|
||||
if Path(__file__).resolve().parent != Path("/backup/scripts"):
|
||||
return None
|
||||
if RUNTIME_LOCK.is_symlink():
|
||||
raise RuntimeError("backup_runtime_gate_unsafe")
|
||||
handle = RUNTIME_LOCK.open("a+")
|
||||
inherited_fd_path: Path | None = None
|
||||
proc_fd_root = Path(f"/proc/{os.getpid()}/fd")
|
||||
if proc_fd_root.is_dir():
|
||||
candidate = proc_fd_root / "197"
|
||||
if candidate.exists() or candidate.is_symlink():
|
||||
inherited_fd_path = candidate
|
||||
else:
|
||||
candidate = Path("/dev/fd/197")
|
||||
if candidate.exists() or candidate.is_symlink():
|
||||
inherited_fd_path = candidate
|
||||
|
||||
if inherited_fd_path is None:
|
||||
handle = RUNTIME_LOCK.open("a+")
|
||||
else:
|
||||
if not RUNTIME_LOCK.is_file():
|
||||
raise RuntimeError("backup_runtime_gate_unavailable")
|
||||
if proc_fd_root.is_dir():
|
||||
try:
|
||||
inherited_target = inherited_fd_path.resolve(strict=True)
|
||||
expected_target = RUNTIME_LOCK.resolve(strict=True)
|
||||
except OSError as exc:
|
||||
raise RuntimeError("backup_runtime_inherited_fd_unresolved") from exc
|
||||
else:
|
||||
try:
|
||||
raw_target = fcntl.fcntl(197, getattr(fcntl, "F_GETPATH", 50), b"\0" * 1024)
|
||||
inherited_target = Path(os.fsdecode(raw_target.split(b"\0", 1)[0])).resolve(strict=True)
|
||||
expected_target = RUNTIME_LOCK.resolve(strict=True)
|
||||
except OSError as exc:
|
||||
raise RuntimeError("backup_runtime_inherited_fd_unresolved") from exc
|
||||
if inherited_target != expected_target:
|
||||
raise RuntimeError("backup_runtime_inherited_fd_mismatch")
|
||||
try:
|
||||
handle = os.fdopen(os.dup(197), "a+")
|
||||
except OSError as exc:
|
||||
raise RuntimeError("backup_runtime_inherited_fd_unavailable") from exc
|
||||
|
||||
if RUNTIME_LOCK.stat().st_uid != os.getuid():
|
||||
handle.close()
|
||||
raise RuntimeError("backup_runtime_gate_owner_invalid")
|
||||
|
||||
@@ -0,0 +1,409 @@
|
||||
$ErrorActionPreference = "Stop"
|
||||
|
||||
function Write-ReplayTrace {
|
||||
param([string]$Step)
|
||||
if ($ReplayTrace) { Write-Output "REPLAY_STEP=$Step" }
|
||||
}
|
||||
|
||||
if (-not $BrokerGzipBase64) { throw "broker_payload_missing" }
|
||||
Write-ReplayTrace "decode_source"
|
||||
$compressed = [Convert]::FromBase64String([string]$BrokerGzipBase64)
|
||||
$inputStream = New-Object IO.MemoryStream(,$compressed)
|
||||
$gzip = New-Object IO.Compression.GzipStream(
|
||||
$inputStream,
|
||||
[IO.Compression.CompressionMode]::Decompress
|
||||
)
|
||||
$reader = New-Object IO.StreamReader($gzip, [Text.Encoding]::UTF8)
|
||||
try {
|
||||
$brokerSource = $reader.ReadToEnd()
|
||||
} finally {
|
||||
$reader.Dispose()
|
||||
$gzip.Dispose()
|
||||
$inputStream.Dispose()
|
||||
}
|
||||
|
||||
$tokens = $null
|
||||
$parseErrors = $null
|
||||
$ast = [Management.Automation.Language.Parser]::ParseInput(
|
||||
$brokerSource,
|
||||
[ref]$tokens,
|
||||
[ref]$parseErrors
|
||||
)
|
||||
if ($parseErrors.Count -ne 0) { throw "broker_parse_failed" }
|
||||
Write-ReplayTrace "define_functions"
|
||||
|
||||
$functionNames = @(
|
||||
"Test-Agent99JsonField",
|
||||
"Assert-Agent99JsonFields",
|
||||
"Assert-Agent99ExactJsonFields",
|
||||
"Convert-Agent99ExecutorDocument",
|
||||
"Assert-Agent99ApplyResultFields",
|
||||
"Assert-Agent99VerifierFields",
|
||||
"Get-Agent99ExecutorResult",
|
||||
"Get-Agent99CommittedFailureResult",
|
||||
"Assert-Agent99VerifierEvidence",
|
||||
"Assert-Agent99PreflightEvidence",
|
||||
"Get-Agent99TransportReceiptReadback",
|
||||
"New-Agent99TransportLossReconciliation",
|
||||
"Assert-Agent99ExecutorEvidence",
|
||||
"Assert-Agent99ReconciliationEvidence",
|
||||
"Assert-Agent99ExistingBrokerEvidence"
|
||||
)
|
||||
foreach ($functionName in $functionNames) {
|
||||
$node = $ast.Find(
|
||||
{
|
||||
param($candidate)
|
||||
$candidate -is [Management.Automation.Language.FunctionDefinitionAst] -and
|
||||
$candidate.Name -eq $functionName
|
||||
},
|
||||
$true
|
||||
)
|
||||
if ($null -eq $node) { throw "broker_function_missing_$functionName" }
|
||||
Invoke-Expression $node.Extent.Text
|
||||
}
|
||||
|
||||
$SourceRevision = "a" * 40
|
||||
$RunId = "windows99-contract-replay"
|
||||
$Mode = "Apply"
|
||||
$TargetHost = "192.168.0.110"
|
||||
$ExpectedFileCount = 18
|
||||
$sourceHead = "b" * 40
|
||||
$VerifierSha256 = "c" * 64
|
||||
$payloadReceipt = "/backup/status/agent99-backup-runtime-deploy-$RunId.json"
|
||||
$terminalReceipt = "/backup/status/agent99-backup-runtime-terminal-$RunId.json"
|
||||
|
||||
function Copy-ReplayDocument {
|
||||
param([object]$Document)
|
||||
return ($Document | ConvertTo-Json -Compress -Depth 12 | ConvertFrom-Json)
|
||||
}
|
||||
|
||||
function New-ReplayTransport {
|
||||
param([object]$Document, [int]$ExitCode)
|
||||
return [pscustomobject]@{
|
||||
ok = [bool]($ExitCode -eq 0)
|
||||
exitCode = $ExitCode
|
||||
reason = if ($ExitCode -eq 0) { "completed" } else { "process_failed" }
|
||||
stdout = $Document | ConvertTo-Json -Compress -Depth 12
|
||||
stderrPresent = $false
|
||||
}
|
||||
}
|
||||
|
||||
function Assert-Rejected {
|
||||
param([scriptblock]$Operation, [string]$Name)
|
||||
$accepted = $false
|
||||
try {
|
||||
& $Operation | Out-Null
|
||||
$accepted = $true
|
||||
} catch {}
|
||||
if ($accepted) { throw "contract_mutation_accepted_$Name" }
|
||||
}
|
||||
|
||||
$verifier = [pscustomobject]@{
|
||||
schemaVersion = "agent99_host110_backup_runtime_verifier_v1"
|
||||
ok = $true
|
||||
sourceRevision = $SourceRevision
|
||||
runId = $RunId
|
||||
fileCount = 18
|
||||
backupScriptFileCount = 17
|
||||
backupHealthExporterIncluded = $true
|
||||
destination = "/backup/scripts"
|
||||
exporterDestination = "/home/wooo/scripts/backup-health-textfile-exporter.py"
|
||||
remoteWritePerformed = $false
|
||||
secretValuesRead = $false
|
||||
selfIdentityVerified = $true
|
||||
verifierSha256 = $VerifierSha256
|
||||
}
|
||||
$success = [pscustomobject]@{
|
||||
schemaVersion = "agent99_host110_backup_runtime_executor_v1"
|
||||
mode = "apply"
|
||||
status = "verified"
|
||||
ok = $true
|
||||
sourceRevision = $SourceRevision
|
||||
sourceHead = $sourceHead
|
||||
runId = $RunId
|
||||
fileCount = 18
|
||||
backupScriptFileCount = 17
|
||||
backupHealthExporterIncluded = $true
|
||||
payloadCommitted = $true
|
||||
rollbackAttempted = $false
|
||||
rollbackPerformed = $false
|
||||
rollbackVerified = $false
|
||||
zeroResidueVerified = $true
|
||||
zeroResidueScope = "run_owned_destination_temps"
|
||||
stageCleanupVerified = $true
|
||||
rollbackPrestateCleanupVerified = $true
|
||||
receipt = $payloadReceipt
|
||||
terminalReceipt = $terminalReceipt
|
||||
terminalReceiptPublished = $true
|
||||
terminalExitCode = 0
|
||||
deferredSignal = $null
|
||||
independentVerifierVerified = $true
|
||||
verifier = $verifier
|
||||
}
|
||||
|
||||
Write-ReplayTrace "accept_success"
|
||||
Get-Agent99ExecutorResult (New-ReplayTransport $success 0) "Apply" $VerifierSha256 | Out-Null
|
||||
$topLevelRequired = @(
|
||||
"schemaVersion", "mode", "status", "ok", "sourceRevision", "sourceHead",
|
||||
"runId", "fileCount", "backupScriptFileCount", "backupHealthExporterIncluded",
|
||||
"payloadCommitted", "rollbackAttempted", "rollbackPerformed", "rollbackVerified",
|
||||
"zeroResidueVerified", "zeroResidueScope", "stageCleanupVerified",
|
||||
"rollbackPrestateCleanupVerified", "receipt", "terminalReceipt", "terminalExitCode",
|
||||
"terminalReceiptPublished", "deferredSignal", "independentVerifierVerified", "verifier"
|
||||
)
|
||||
$verifierRequired = @(
|
||||
"schemaVersion", "ok", "sourceRevision", "runId", "fileCount",
|
||||
"backupScriptFileCount", "backupHealthExporterIncluded", "remoteWritePerformed",
|
||||
"secretValuesRead", "selfIdentityVerified", "destination", "exporterDestination",
|
||||
"verifierSha256"
|
||||
)
|
||||
$missingFieldRejections = 0
|
||||
Write-ReplayTrace "reject_missing_top"
|
||||
foreach ($field in $topLevelRequired) {
|
||||
$mutated = Copy-ReplayDocument $success
|
||||
$mutated.PSObject.Properties.Remove($field)
|
||||
Assert-Rejected { Get-Agent99ExecutorResult (New-ReplayTransport $mutated 0) "Apply" $VerifierSha256 } "missing_$field"
|
||||
$missingFieldRejections++
|
||||
}
|
||||
Write-ReplayTrace "reject_missing_verifier"
|
||||
foreach ($field in $verifierRequired) {
|
||||
$mutated = Copy-ReplayDocument $success
|
||||
$mutated.verifier.PSObject.Properties.Remove($field)
|
||||
Assert-Rejected { Get-Agent99ExecutorResult (New-ReplayTransport $mutated 0) "Apply" $VerifierSha256 } "missing_verifier_$field"
|
||||
$missingFieldRejections++
|
||||
}
|
||||
|
||||
$typeMutations = @(
|
||||
@{ field = "rollbackAttempted"; value = "false" },
|
||||
@{ field = "rollbackPerformed"; value = "false" },
|
||||
@{ field = "rollbackVerified"; value = "false" },
|
||||
@{ field = "terminalExitCode"; value = "0" },
|
||||
@{ field = "deferredSignal"; value = 0 }
|
||||
)
|
||||
$typeRejections = 0
|
||||
Write-ReplayTrace "reject_types"
|
||||
foreach ($mutation in $typeMutations) {
|
||||
$mutated = Copy-ReplayDocument $success
|
||||
$mutated.($mutation.field) = $mutation.value
|
||||
Assert-Rejected { Get-Agent99ExecutorResult (New-ReplayTransport $mutated 0) "Apply" $VerifierSha256 } "type_$($mutation.field)"
|
||||
$typeRejections++
|
||||
}
|
||||
$mutated = Copy-ReplayDocument $success
|
||||
$mutated.verifier.remoteWritePerformed = "false"
|
||||
Assert-Rejected { Get-Agent99ExecutorResult (New-ReplayTransport $mutated 0) "Apply" $VerifierSha256 } "type_verifier_remoteWritePerformed"
|
||||
$typeRejections++
|
||||
|
||||
$cleanupPending = Copy-ReplayDocument $success
|
||||
$cleanupPending.ok = $false
|
||||
$cleanupPending.status = "cleanup_pending"
|
||||
$cleanupPending.stageCleanupVerified = $false
|
||||
$cleanupPending.terminalExitCode = 75
|
||||
$cleanupPending.verifier = $null
|
||||
Write-ReplayTrace "accept_cleanup_pending"
|
||||
Get-Agent99CommittedFailureResult (New-ReplayTransport $cleanupPending 75) | Out-Null
|
||||
|
||||
$signalDeferred = Copy-ReplayDocument $success
|
||||
$signalDeferred.ok = $false
|
||||
$signalDeferred.status = "committed_signal_deferred"
|
||||
$signalDeferred.terminalExitCode = 143
|
||||
$signalDeferred.deferredSignal = "TERM"
|
||||
$signalDeferred.verifier = $null
|
||||
Write-ReplayTrace "accept_signal_deferred"
|
||||
Get-Agent99CommittedFailureResult (New-ReplayTransport $signalDeferred 143) | Out-Null
|
||||
|
||||
$terminalReceiptFailed = Copy-ReplayDocument $success
|
||||
$terminalReceiptFailed.ok = $false
|
||||
$terminalReceiptFailed.status = "terminal_receipt_failed"
|
||||
$terminalReceiptFailed.terminalReceiptPublished = $false
|
||||
$terminalReceiptFailed.terminalExitCode = 76
|
||||
$terminalReceiptFailed.verifier = $null
|
||||
Write-ReplayTrace "accept_terminal_receipt_failed"
|
||||
Get-Agent99CommittedFailureResult (New-ReplayTransport $terminalReceiptFailed 76) | Out-Null
|
||||
|
||||
$EvidencePath = "C:\Wooo\Agent99\evidence\host110-backup-runtime\agent99-host110-backup-runtime-$RunId.json"
|
||||
$preflight = [pscustomobject]@{
|
||||
schemaVersion = "agent99_host110_backup_runtime_preflight_v1"
|
||||
ok = $true
|
||||
sourceRevision = $SourceRevision
|
||||
sourceHead = $sourceHead
|
||||
fileCount = 18
|
||||
backupScriptFileCount = 17
|
||||
backupHealthExporterIncluded = $true
|
||||
remoteWritePerformed = $false
|
||||
}
|
||||
$brokerSuccess = [pscustomobject]@{
|
||||
schemaVersion = "agent99_host110_backup_runtime_broker_receipt_v3"
|
||||
ok = $true
|
||||
status = "verified"
|
||||
mode = "Apply"
|
||||
sourceRevision = $SourceRevision
|
||||
sourceHead = $sourceHead
|
||||
runId = $RunId
|
||||
targetHost = $TargetHost
|
||||
sourceTransport = "windows99_gitea_exact_revision_manifest"
|
||||
executor = "host110_backup_runtime_executor"
|
||||
verifier = "independent_host110_backup_runtime_python_readback"
|
||||
decisionProvider = "deterministic_only"
|
||||
criticProvider = "deterministic_only"
|
||||
agentAction = "controlled_apply"
|
||||
check = $preflight
|
||||
apply = $success
|
||||
verify = $verifier
|
||||
cleanupVerified = $true
|
||||
errorCode = ""
|
||||
elapsedSeconds = 1.25
|
||||
evidence = $EvidencePath
|
||||
secretValuesRead = $false
|
||||
rawSessionStored = $false
|
||||
}
|
||||
Write-ReplayTrace "accept_existing_success"
|
||||
Assert-Agent99ExistingBrokerEvidence $brokerSuccess $EvidencePath
|
||||
|
||||
$brokerRequired = @(
|
||||
"schemaVersion", "ok", "status", "mode", "sourceRevision", "sourceHead",
|
||||
"runId", "targetHost", "sourceTransport", "executor", "verifier",
|
||||
"decisionProvider", "criticProvider", "agentAction", "check", "apply",
|
||||
"verify", "cleanupVerified", "errorCode", "elapsedSeconds", "evidence",
|
||||
"secretValuesRead", "rawSessionStored"
|
||||
)
|
||||
$existingEvidenceRejections = 0
|
||||
Write-ReplayTrace "reject_existing_missing"
|
||||
foreach ($field in $brokerRequired) {
|
||||
$mutated = Copy-ReplayDocument $brokerSuccess
|
||||
$mutated.PSObject.Properties.Remove($field)
|
||||
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_missing_$field"
|
||||
$existingEvidenceRejections++
|
||||
}
|
||||
$existingMutations = @(
|
||||
@{ name = "ok_string_false"; field = "ok"; value = "false" },
|
||||
@{ name = "wrong_mode"; field = "mode"; value = "Verify" },
|
||||
@{ name = "wrong_run_id"; field = "runId"; value = "other-run" },
|
||||
@{ name = "wrong_status"; field = "status"; value = "readback_ok" },
|
||||
@{ name = "wrong_cleanup"; field = "cleanupVerified"; value = $false },
|
||||
@{ name = "cleanup_string"; field = "cleanupVerified"; value = "true" },
|
||||
@{ name = "elapsed_string"; field = "elapsedSeconds"; value = "1.25" }
|
||||
)
|
||||
Write-ReplayTrace "reject_existing_conflicts"
|
||||
foreach ($mutation in $existingMutations) {
|
||||
$mutated = Copy-ReplayDocument $brokerSuccess
|
||||
$mutated.($mutation.field) = $mutation.value
|
||||
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_$($mutation.name)"
|
||||
$existingEvidenceRejections++
|
||||
}
|
||||
$mutated = Copy-ReplayDocument $brokerSuccess
|
||||
$mutated.apply.runId = "other-run"
|
||||
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_apply_run_conflict"
|
||||
$existingEvidenceRejections++
|
||||
$mutated = Copy-ReplayDocument $brokerSuccess
|
||||
$mutated.verify.PSObject.Properties.Remove("selfIdentityVerified")
|
||||
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_verify_nested_missing"
|
||||
$existingEvidenceRejections++
|
||||
$mutated = Copy-ReplayDocument $brokerSuccess
|
||||
$mutated | Add-Member -NotePropertyName unexpected -NotePropertyValue $true
|
||||
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_extra_top"
|
||||
$existingEvidenceRejections++
|
||||
$mutated = Copy-ReplayDocument $brokerSuccess
|
||||
$mutated.apply | Add-Member -NotePropertyName unexpected -NotePropertyValue $true
|
||||
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_extra_apply"
|
||||
$existingEvidenceRejections++
|
||||
|
||||
$brokerFailure = Copy-ReplayDocument $brokerSuccess
|
||||
$brokerFailure.ok = $false
|
||||
$brokerFailure.status = "failed"
|
||||
$brokerFailure.cleanupVerified = $false
|
||||
$brokerFailure.errorCode = "host110_apply_cleanup_pending"
|
||||
$brokerFailure.apply = $cleanupPending
|
||||
$brokerFailure.verify = $null
|
||||
Write-ReplayTrace "accept_existing_failure"
|
||||
Assert-Agent99ExistingBrokerEvidence $brokerFailure $EvidencePath
|
||||
$failureWithoutPreflight = Copy-ReplayDocument $brokerFailure
|
||||
$failureWithoutPreflight.check = $null
|
||||
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $failureWithoutPreflight $EvidencePath } "failure_preflight_missing"
|
||||
$existingEvidenceRejections++
|
||||
$failureWithImpossibleVerifier = Copy-ReplayDocument $brokerFailure
|
||||
$failureWithImpossibleVerifier.verify = $verifier
|
||||
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $failureWithImpossibleVerifier $EvidencePath } "failure_verifier_conflict"
|
||||
$existingEvidenceRejections++
|
||||
$postcommitCleanupFailure = Copy-ReplayDocument $brokerSuccess
|
||||
$postcommitCleanupFailure.ok = $false
|
||||
$postcommitCleanupFailure.status = "failed"
|
||||
$postcommitCleanupFailure.cleanupVerified = $false
|
||||
$postcommitCleanupFailure.errorCode = "remote_source_stage_cleanup_failed"
|
||||
Assert-Agent99ExistingBrokerEvidence $postcommitCleanupFailure $EvidencePath
|
||||
$malformedFailure = Copy-ReplayDocument $brokerFailure
|
||||
$malformedFailure.ok = "false"
|
||||
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $malformedFailure $EvidencePath } "malformed_failure_not_upgraded"
|
||||
$existingEvidenceRejections++
|
||||
|
||||
$sourcePackage = [pscustomobject]@{ sourceHead = $sourceHead }
|
||||
$transportLoss = [pscustomobject]@{ reason = "transport_timeout"; exitCode = -2 }
|
||||
$receiptReadback = [pscustomobject]@{
|
||||
schemaVersion = "agent99_host110_backup_transport_receipt_readback_v1"
|
||||
ok = $true
|
||||
sourceRevision = $SourceRevision
|
||||
sourceHead = $sourceHead
|
||||
runId = $RunId
|
||||
payloadReceipt = $payloadReceipt
|
||||
payloadState = "valid"
|
||||
terminalReceipt = $terminalReceipt
|
||||
terminalState = "missing"
|
||||
terminalStatus = ""
|
||||
terminalCleanupVerified = $false
|
||||
terminalExitCode = -1
|
||||
remoteWritePerformed = $false
|
||||
secretValuesRead = $false
|
||||
}
|
||||
Write-ReplayTrace "reconcile_terminal_missing"
|
||||
$terminalMissing = New-Agent99TransportLossReconciliation $transportLoss $sourcePackage $receiptReadback $verifier
|
||||
if (
|
||||
$terminalMissing.status -ne "committed_verified_terminal_missing" -or
|
||||
-not $terminalMissing.payloadCommitted -or
|
||||
-not $terminalMissing.destinationVerified -or
|
||||
$terminalMissing.retryApplyAllowed
|
||||
) { throw "transport_terminal_missing_reconciliation_failed" }
|
||||
Assert-Agent99ReconciliationEvidence $terminalMissing $sourceHead
|
||||
|
||||
$terminalPresentReadback = Copy-ReplayDocument $receiptReadback
|
||||
$terminalPresentReadback.terminalState = "valid"
|
||||
$terminalPresentReadback.terminalStatus = "verified"
|
||||
$terminalPresentReadback.terminalCleanupVerified = $true
|
||||
$terminalPresentReadback.terminalExitCode = 0
|
||||
$transportRecovered = New-Agent99TransportLossReconciliation $transportLoss $sourcePackage $terminalPresentReadback $verifier
|
||||
if ($transportRecovered.status -ne "committed_verified_transport_lost" -or $transportRecovered.retryApplyAllowed) {
|
||||
throw "transport_verified_reconciliation_failed"
|
||||
}
|
||||
Assert-Agent99ReconciliationEvidence $transportRecovered $sourceHead
|
||||
$conflictingTransport = Copy-ReplayDocument $transportRecovered
|
||||
$conflictingTransport.terminalStatus = "cleanup_pending"
|
||||
Assert-Rejected { Assert-Agent99ReconciliationEvidence $conflictingTransport $sourceHead } "transport_terminal_status_conflict"
|
||||
$existingEvidenceRejections++
|
||||
|
||||
$committedUnknown = New-Agent99TransportLossReconciliation `
|
||||
$transportLoss $sourcePackage $null $null "receipt_readback_unavailable" "verifier_unavailable"
|
||||
if (
|
||||
$committedUnknown.status -ne "committed_unknown" -or
|
||||
$committedUnknown.payloadCommitted -or
|
||||
$committedUnknown.destinationVerified -or
|
||||
$committedUnknown.retryApplyAllowed
|
||||
) { throw "transport_unknown_reconciliation_failed" }
|
||||
Assert-Agent99ReconciliationEvidence $committedUnknown $sourceHead
|
||||
|
||||
Write-ReplayTrace "complete"
|
||||
[pscustomobject]@{
|
||||
schemaVersion = "agent99_host110_broker_contract_replay_v1"
|
||||
ok = $true
|
||||
parser = "WindowsPowerShell"
|
||||
successAccepted = $true
|
||||
cleanupPendingPreserved = $true
|
||||
committedSignalPreserved = $true
|
||||
terminalReceiptFailurePreserved = $true
|
||||
existingSuccessAccepted = $true
|
||||
existingFailurePreserved = $true
|
||||
existingEvidenceRejections = $existingEvidenceRejections
|
||||
terminalMissingReconciled = $true
|
||||
committedUnknownPreserved = $true
|
||||
missingFieldRejections = $missingFieldRejections
|
||||
typeRejections = $typeRejections
|
||||
remoteWritePerformed = $false
|
||||
secretValuesRead = $false
|
||||
} | ConvertTo-Json -Compress
|
||||
@@ -1,6 +1,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import fcntl
|
||||
import grp
|
||||
import hashlib
|
||||
import importlib.util
|
||||
@@ -20,6 +21,9 @@ ROOT = Path(__file__).resolve().parents[3]
|
||||
BROKER = ROOT / "agent99-host110-backup-runtime-broker.ps1"
|
||||
EXECUTOR = ROOT / "scripts" / "backup" / "host110-backup-runtime-executor.sh"
|
||||
VERIFIER = ROOT / "scripts" / "backup" / "verify-host110-backup-runtime.py"
|
||||
BROKER_CONTRACT_REPLAY = (
|
||||
ROOT / "scripts" / "reboot-recovery" / "tests" / "agent99-host110-broker-contract-replay.ps1"
|
||||
)
|
||||
|
||||
|
||||
EXPECTED_RUNTIME_FILES = {
|
||||
@@ -71,6 +75,12 @@ def _powershell_array(source: str, variable: str) -> tuple[str, ...]:
|
||||
return tuple(re.findall(r'"([^"]+)"', match.group(1)))
|
||||
|
||||
|
||||
def _powershell_function(source: str, name: str) -> str:
|
||||
start = source.index(f"function {name} {{")
|
||||
next_function = source.find("\nfunction ", start + 1)
|
||||
return source[start:] if next_function < 0 else source[start:next_function]
|
||||
|
||||
|
||||
def _executor_payload_order(source: str) -> tuple[str, ...]:
|
||||
match = re.search(r"readonly -a RUNTIME_FILES=\((.*?)\n\)", source, re.DOTALL)
|
||||
assert match is not None
|
||||
@@ -85,6 +95,88 @@ def _write_executable(path: Path, source: str) -> None:
|
||||
path.chmod(0o755)
|
||||
|
||||
|
||||
def _write_fcntl_flock(path: Path) -> None:
|
||||
_write_executable(
|
||||
path,
|
||||
"""#!/usr/bin/env python3
|
||||
import fcntl
|
||||
import sys
|
||||
import time
|
||||
|
||||
args = sys.argv[1:]
|
||||
operation = fcntl.LOCK_SH if "-s" in args else fcntl.LOCK_EX
|
||||
wait_seconds = float(args[args.index("-w") + 1]) if "-w" in args else 0.0
|
||||
descriptor = int(args[-1])
|
||||
deadline = time.monotonic() + wait_seconds
|
||||
while True:
|
||||
try:
|
||||
fcntl.flock(descriptor, operation | fcntl.LOCK_NB)
|
||||
raise SystemExit(0)
|
||||
except BlockingIOError:
|
||||
if time.monotonic() >= deadline:
|
||||
raise SystemExit(1)
|
||||
time.sleep(0.02)
|
||||
""",
|
||||
)
|
||||
|
||||
|
||||
def _build_shared_lock_gate_fixture(
|
||||
tmp_path: Path,
|
||||
entrypoint: str,
|
||||
*,
|
||||
reject_fd_reopen: bool = False,
|
||||
body_sleep_seconds: float = 0.0,
|
||||
) -> tuple[Path, Path, Path, dict[str, str]]:
|
||||
runtime_root = tmp_path / "runtime-scripts"
|
||||
runtime_root.mkdir()
|
||||
lock_path = tmp_path / "runtime.lock"
|
||||
fake_bin = tmp_path / "fake-bin"
|
||||
fake_bin.mkdir()
|
||||
_write_fcntl_flock(fake_bin / "flock")
|
||||
|
||||
source_map = {
|
||||
"common": ROOT / "scripts/backup/common.sh",
|
||||
"restore": ROOT / "scripts/backup/gitea-full-backup-restore-drill.sh",
|
||||
"offsite": ROOT / "scripts/backup/backup-offsite-readiness-gate.sh",
|
||||
}
|
||||
source = source_map[entrypoint].read_text(encoding="utf-8")
|
||||
source = source.replace("/backup/scripts", str(runtime_root))
|
||||
source = source.replace("/tmp/agent99-host110-backup-runtime.lock", str(lock_path))
|
||||
source = source.replace("flock -s -w 60", "flock -s -w 1")
|
||||
if reject_fd_reopen:
|
||||
source, replacement_count = re.subn(
|
||||
r"(?m)^(\s*)exec 197>>.*$",
|
||||
r"\1printf 'FD_REOPENED=1\\n' >&2; exit 98",
|
||||
source,
|
||||
)
|
||||
assert replacement_count == 1
|
||||
|
||||
body = "printf 'BODY_REACHED=1\\n'\n"
|
||||
if body_sleep_seconds:
|
||||
body += f"sleep {body_sleep_seconds}\n"
|
||||
if entrypoint == "common":
|
||||
common_path = runtime_root / "common.sh"
|
||||
gate_source = source.split("# WOOO AIOps - 備份共用函式庫", 1)[0]
|
||||
_write_executable(common_path, gate_source)
|
||||
script_path = runtime_root / "backup-gitea.sh"
|
||||
_write_executable(
|
||||
script_path,
|
||||
"#!/usr/bin/env bash\n"
|
||||
"set -euo pipefail\n"
|
||||
f"source {str(common_path)!r}\n"
|
||||
+ body,
|
||||
)
|
||||
else:
|
||||
product_start = "DUMP_ZIP=" if entrypoint == "restore" else "BACKUP_BASE="
|
||||
gate_source = source.split(product_start, 1)[0]
|
||||
script_path = runtime_root / source_map[entrypoint].name
|
||||
_write_executable(script_path, gate_source + body)
|
||||
|
||||
environment = os.environ.copy()
|
||||
environment["PATH"] = f"{fake_bin}:{environment['PATH']}"
|
||||
return script_path, runtime_root, lock_path, environment
|
||||
|
||||
|
||||
def _build_executor_harness(tmp_path: Path, run_id: str) -> dict[str, object]:
|
||||
destination = tmp_path / "backup-scripts"
|
||||
exporter_root = tmp_path / "exporter"
|
||||
@@ -240,6 +332,10 @@ def _run_executor_harness(harness: dict[str, object], **extra_env: str) -> subpr
|
||||
"HOST110_BACKUP_RUNTIME_TEST_RECEIPT_FAULT",
|
||||
"HOST110_BACKUP_RUNTIME_TEST_APPLY_FAULT",
|
||||
"HOST110_BACKUP_RUNTIME_TEST_ROLLBACK_FAULT",
|
||||
"HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL",
|
||||
"HOST110_BACKUP_RUNTIME_TEST_TERMINAL_SIGNAL",
|
||||
"HOST110_BACKUP_RUNTIME_TEST_TERMINAL_RECEIPT_FAULT",
|
||||
"HOST110_BACKUP_RUNTIME_TEST_CLEANUP_FAULT",
|
||||
):
|
||||
environment.pop(name, None)
|
||||
environment["HOST110_BACKUP_RUNTIME_ENABLE_TEST_FAULTS"] = "1"
|
||||
@@ -281,6 +377,16 @@ def _assert_prestate_restored(harness: dict[str, object]) -> None:
|
||||
assert (target.read_bytes(), target.stat().st_mode & 0o777) == before_state[name]
|
||||
|
||||
|
||||
def _assert_payload_committed(harness: dict[str, object]) -> None:
|
||||
destination = Path(harness["destination"])
|
||||
exporter_root = Path(harness["exporter_root"])
|
||||
source_stage = Path(harness["source_stage"])
|
||||
for name in harness["payload_order"]:
|
||||
target = exporter_root / name if name == "backup-health-textfile-exporter.py" else destination / name
|
||||
assert target.read_bytes() == (source_stage / name).read_bytes()
|
||||
assert target.stat().st_mode & 0o777 == 0o755
|
||||
|
||||
|
||||
def test_windows99_broker_fetches_exact_gitea_revision_and_is_bounded() -> None:
|
||||
source = BROKER.read_text(encoding="utf-8")
|
||||
|
||||
@@ -304,6 +410,288 @@ def test_windows99_broker_fetches_exact_gitea_revision_and_is_bounded() -> None:
|
||||
assert "taskkill.exe /PID $process.Id /T /F" in source
|
||||
assert "$process.Kill()" in source
|
||||
assert 'Invoke-Agent99BoundedScp $sourcePackage.localPaths' in source
|
||||
assert '[string]$parsed.status -ne "verified"' in source
|
||||
assert '-not [bool]$parsed.payloadCommitted' in source
|
||||
assert '-not [bool]$parsed.stageCleanupVerified' in source
|
||||
assert '-not [bool]$parsed.rollbackPrestateCleanupVerified' in source
|
||||
assert '[string]$parsed.zeroResidueScope -ne "run_owned_destination_temps"' in source
|
||||
assert '[int64]$parsed.terminalExitCode -ne 0' in source
|
||||
assert 'agent99-backup-runtime-terminal-$RunId.json' in source
|
||||
assert '$property = $Object.PSObject.Properties[$Name]' in source
|
||||
assert 'if ($null -eq $property) { return $false }' in source
|
||||
assert 'Get-Agent99CommittedFailureResult $applyTransport' in source
|
||||
assert 'Invoke-Agent99TransportLossReconciliation $applyTransport $sourcePackage' in source
|
||||
assert 'status = "committed_unknown"' in source
|
||||
assert '"committed_verified_terminal_missing"' in source
|
||||
assert 'retryApplyAllowed = $false' in source
|
||||
assert 'throw "host110_apply_$([string]$apply.status)"' in source
|
||||
|
||||
|
||||
def test_broker_required_field_contract_rejects_missing_fields_before_cast() -> None:
|
||||
source = BROKER.read_text(encoding="utf-8")
|
||||
field_guard = _powershell_function(source, "Test-Agent99JsonField")
|
||||
common_contract = _powershell_function(source, "Convert-Agent99ExecutorDocument")
|
||||
apply_contract = _powershell_function(source, "Assert-Agent99ApplyResultFields")
|
||||
verifier_contract = _powershell_function(source, "Assert-Agent99VerifierFields")
|
||||
committed_failure = _powershell_function(source, "Get-Agent99CommittedFailureResult")
|
||||
|
||||
assert '$property = $Object.PSObject.Properties[$Name]' in field_guard
|
||||
assert 'if ($null -eq $property) { return $false }' in field_guard
|
||||
for field in (
|
||||
"schemaVersion",
|
||||
"mode",
|
||||
"ok",
|
||||
"sourceRevision",
|
||||
"sourceHead",
|
||||
"runId",
|
||||
"fileCount",
|
||||
"backupScriptFileCount",
|
||||
"backupHealthExporterIncluded",
|
||||
):
|
||||
assert re.search(rf"^\s+{field} = \"", common_contract, re.MULTILINE), field
|
||||
for field in (
|
||||
"status",
|
||||
"payloadCommitted",
|
||||
"rollbackAttempted",
|
||||
"rollbackPerformed",
|
||||
"rollbackVerified",
|
||||
"zeroResidueVerified",
|
||||
"zeroResidueScope",
|
||||
"stageCleanupVerified",
|
||||
"rollbackPrestateCleanupVerified",
|
||||
"receipt",
|
||||
"terminalReceipt",
|
||||
"terminalReceiptPublished",
|
||||
"terminalExitCode",
|
||||
"deferredSignal",
|
||||
"independentVerifierVerified",
|
||||
"verifier",
|
||||
):
|
||||
assert re.search(rf"^\s+{field} = ", apply_contract, re.MULTILINE), field
|
||||
for field in (
|
||||
"schemaVersion",
|
||||
"ok",
|
||||
"sourceRevision",
|
||||
"runId",
|
||||
"fileCount",
|
||||
"backupScriptFileCount",
|
||||
"backupHealthExporterIncluded",
|
||||
"remoteWritePerformed",
|
||||
"selfIdentityVerified",
|
||||
):
|
||||
assert re.search(rf"^\s+{field} = \"", verifier_contract, re.MULTILINE), field
|
||||
assert 'status -eq "cleanup_pending"' in committed_failure
|
||||
assert 'status -eq "committed_signal_deferred"' in committed_failure
|
||||
assert 'verifier = $VerifierKind' in apply_contract
|
||||
|
||||
|
||||
def test_windows99_contract_replay_uses_actual_broker_functions_without_remote_write() -> None:
|
||||
source = BROKER_CONTRACT_REPLAY.read_text(encoding="utf-8")
|
||||
|
||||
assert "FunctionDefinitionAst" in source
|
||||
for function_name in (
|
||||
"Test-Agent99JsonField",
|
||||
"Assert-Agent99JsonFields",
|
||||
"Assert-Agent99ExactJsonFields",
|
||||
"Convert-Agent99ExecutorDocument",
|
||||
"Assert-Agent99ApplyResultFields",
|
||||
"Assert-Agent99VerifierFields",
|
||||
"Get-Agent99ExecutorResult",
|
||||
"Get-Agent99CommittedFailureResult",
|
||||
"Assert-Agent99VerifierEvidence",
|
||||
"Assert-Agent99PreflightEvidence",
|
||||
"Get-Agent99TransportReceiptReadback",
|
||||
"New-Agent99TransportLossReconciliation",
|
||||
"Assert-Agent99ExecutorEvidence",
|
||||
"Assert-Agent99ReconciliationEvidence",
|
||||
"Assert-Agent99ExistingBrokerEvidence",
|
||||
):
|
||||
assert f'"{function_name}"' in source
|
||||
assert "$mutated.PSObject.Properties.Remove($field)" in source
|
||||
assert "$mutated.verifier.PSObject.Properties.Remove($field)" in source
|
||||
assert 'status = "cleanup_pending"' in source
|
||||
assert 'status = "committed_signal_deferred"' in source
|
||||
assert 'status = "terminal_receipt_failed"' in source
|
||||
assert 'ok = "false"' in source
|
||||
assert 'status -ne "committed_verified_terminal_missing"' in source
|
||||
assert 'status -ne "committed_unknown"' in source
|
||||
assert "existingEvidenceRejections" in source
|
||||
assert "malformed_failure_not_upgraded" in source
|
||||
assert "remoteWritePerformed = $false" in source
|
||||
assert "secretValuesRead = $false" in source
|
||||
for forbidden in ("New-Item", "Set-Content", "WriteAllText", "ssh.exe", "scp.exe"):
|
||||
assert forbidden not in source
|
||||
|
||||
|
||||
def test_existing_broker_evidence_replay_is_exact_typed_and_nested_fail_closed() -> None:
|
||||
source = BROKER.read_text(encoding="utf-8")
|
||||
exact_guard = _powershell_function(source, "Assert-Agent99ExactJsonFields")
|
||||
existing = _powershell_function(source, "Assert-Agent99ExistingBrokerEvidence")
|
||||
reconciliation = _powershell_function(source, "Assert-Agent99ReconciliationEvidence")
|
||||
|
||||
assert "$properties.Count -ne $Contract.Count" in exact_guard
|
||||
assert "$Contract.ContainsKey([string]$property.Name)" in exact_guard
|
||||
for field in (
|
||||
"schemaVersion",
|
||||
"ok",
|
||||
"status",
|
||||
"mode",
|
||||
"sourceRevision",
|
||||
"sourceHead",
|
||||
"runId",
|
||||
"targetHost",
|
||||
"sourceTransport",
|
||||
"executor",
|
||||
"verifier",
|
||||
"decisionProvider",
|
||||
"criticProvider",
|
||||
"agentAction",
|
||||
"check",
|
||||
"apply",
|
||||
"verify",
|
||||
"cleanupVerified",
|
||||
"errorCode",
|
||||
"elapsedSeconds",
|
||||
"evidence",
|
||||
"secretValuesRead",
|
||||
"rawSessionStored",
|
||||
):
|
||||
assert re.search(rf"^\s+{field} = \"", existing, re.MULTILINE), field
|
||||
assert "Assert-Agent99PreflightEvidence $Existing.check" in existing
|
||||
assert "Assert-Agent99ExecutorEvidence $Existing.apply" in existing
|
||||
assert "Assert-Agent99ReconciliationEvidence $Existing.apply" in existing
|
||||
assert "Assert-Agent99VerifierEvidence $Existing.verify" in existing
|
||||
assert 'throw "existing_broker_apply_failure_evidence_missing"' in existing
|
||||
assert 'retryApplyAllowed = "Boolean"' in reconciliation
|
||||
assert "host110_existing_reconciliation_payload_truth_mismatch" in reconciliation
|
||||
|
||||
|
||||
def test_transport_loss_reconciliation_is_read_only_run_bound_and_never_retries_apply() -> None:
|
||||
source = BROKER.read_text(encoding="utf-8")
|
||||
reader = source.split("$TransportReconciliationReadback = @'", 1)[1].split("\n'@", 1)[0]
|
||||
reconcile = _powershell_function(source, "Invoke-Agent99TransportLossReconciliation")
|
||||
classify = _powershell_function(source, "New-Agent99TransportLossReconciliation")
|
||||
|
||||
for expected in (
|
||||
"agent99-backup-runtime-deploy-{args.run_id}.json",
|
||||
"agent99-backup-runtime-terminal-{args.run_id}.json",
|
||||
'set(document) == set(contract)',
|
||||
'payload["status"] == "payload_verified"',
|
||||
'terminal["payloadReceipt"] == str(payload_path)',
|
||||
'"remoteWritePerformed": False',
|
||||
'"secretValuesRead": False',
|
||||
):
|
||||
assert expected in reader
|
||||
assert "Invoke-Agent99BoundedSsh $command 60" in reconcile
|
||||
assert "Invoke-Agent99NoWriteVerifier $SourcePackage" in reconcile
|
||||
assert 'status = "committed_unknown"' in classify
|
||||
assert '"committed_verified_terminal_missing"' in classify
|
||||
assert '"committed_verified_transport_lost"' in classify
|
||||
assert "retryApplyAllowed = $false" in classify
|
||||
for forbidden in ("rm ", "mv ", "install ", "scp", "--mode apply"):
|
||||
assert forbidden not in reconcile
|
||||
|
||||
|
||||
def test_transport_receipt_readback_rejects_extra_and_wrong_typed_documents(tmp_path: Path) -> None:
|
||||
source = BROKER.read_text(encoding="utf-8")
|
||||
reader = source.split("$TransportReconciliationReadback = @'", 1)[1].split("\n'@", 1)[0]
|
||||
status_root = tmp_path / "status"
|
||||
status_root.mkdir()
|
||||
run_id = "transport-loss-replay"
|
||||
source_revision = "a" * 40
|
||||
source_head = "b" * 40
|
||||
verifier_sha256 = "c" * 64
|
||||
payload_path = status_root / f"agent99-backup-runtime-deploy-{run_id}.json"
|
||||
terminal_path = status_root / f"agent99-backup-runtime-terminal-{run_id}.json"
|
||||
payload = {
|
||||
"schemaVersion": "agent99_host110_backup_runtime_deploy_receipt_v3",
|
||||
"status": "payload_verified",
|
||||
"sourceRevision": source_revision,
|
||||
"sourceHead": source_head,
|
||||
"runId": run_id,
|
||||
"fileCount": 18,
|
||||
"backupScriptFileCount": 17,
|
||||
"backupHealthExporterIncluded": True,
|
||||
"rollbackAttempted": False,
|
||||
"rollbackPerformed": False,
|
||||
"rollbackVerified": False,
|
||||
"zeroResidueVerified": True,
|
||||
"zeroResidueScope": "run_owned_destination_temps",
|
||||
"verifierSha256": verifier_sha256,
|
||||
"executorHost": "192.168.0.110",
|
||||
"productionServiceRestarted": False,
|
||||
"secretValuesRead": False,
|
||||
"writtenAt": 1,
|
||||
}
|
||||
terminal = {
|
||||
"schemaVersion": "agent99_host110_backup_runtime_terminal_receipt_v1",
|
||||
"status": "verified",
|
||||
"ok": True,
|
||||
"payloadCommitted": True,
|
||||
"payloadReceipt": str(payload_path),
|
||||
"sourceRevision": source_revision,
|
||||
"sourceHead": source_head,
|
||||
"runId": run_id,
|
||||
"fileCount": 18,
|
||||
"backupScriptFileCount": 17,
|
||||
"backupHealthExporterIncluded": True,
|
||||
"stageCleanupVerified": True,
|
||||
"rollbackPrestateCleanupVerified": True,
|
||||
"cleanupVerified": True,
|
||||
"independentVerifierVerified": True,
|
||||
"zeroResidueScope": "run_owned_destination_temps_and_internal_stage_prestate",
|
||||
"deferredSignal": None,
|
||||
"terminalExitCode": 0,
|
||||
"verifierSha256": verifier_sha256,
|
||||
"executorHost": "192.168.0.110",
|
||||
"productionServiceRestarted": False,
|
||||
"secretValuesRead": False,
|
||||
"writtenAt": 2,
|
||||
}
|
||||
|
||||
def run_readback() -> dict[str, object]:
|
||||
result = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-",
|
||||
"--status-root",
|
||||
str(status_root),
|
||||
"--source-revision",
|
||||
source_revision,
|
||||
"--source-head",
|
||||
source_head,
|
||||
"--run-id",
|
||||
run_id,
|
||||
"--verifier-sha256",
|
||||
verifier_sha256,
|
||||
],
|
||||
input=reader,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert result.returncode == 0, result.stderr
|
||||
return json.loads(result.stdout)
|
||||
|
||||
payload_path.write_text(json.dumps(payload), encoding="utf-8")
|
||||
terminal_path.write_text(json.dumps(terminal), encoding="utf-8")
|
||||
valid = run_readback()
|
||||
assert valid["payloadState"] == "valid"
|
||||
assert valid["terminalState"] == "valid"
|
||||
assert valid["terminalStatus"] == "verified"
|
||||
|
||||
payload["unexpected"] = True
|
||||
payload_path.write_text(json.dumps(payload), encoding="utf-8")
|
||||
assert run_readback()["payloadState"] == "invalid"
|
||||
payload.pop("unexpected")
|
||||
payload["writtenAt"] = "1"
|
||||
payload_path.write_text(json.dumps(payload), encoding="utf-8")
|
||||
assert run_readback()["payloadState"] == "invalid"
|
||||
payload["writtenAt"] = 1
|
||||
payload_path.write_text(json.dumps(payload), encoding="utf-8")
|
||||
terminal["ok"] = "false"
|
||||
terminal_path.write_text(json.dumps(terminal), encoding="utf-8")
|
||||
assert run_readback()["terminalState"] == "invalid"
|
||||
|
||||
|
||||
def test_broker_scp_basenames_match_executor_source_stage_contract() -> None:
|
||||
@@ -385,11 +773,16 @@ def test_host110_executor_is_manifest_bound_atomic_and_fail_closed() -> None:
|
||||
assert '"rollbackPerformed": rollback_performed' in source
|
||||
assert '"rollbackVerified": rollback_verified' in source
|
||||
assert '"zeroResidueVerified": zero_residue_verified' in source
|
||||
assert '"zeroResidueScope": "run_owned_destination_temps"' in source
|
||||
assert 'cleanup_run_owned_temps' in source
|
||||
assert 'verify_run_owned_temp_absence' in source
|
||||
assert 'HOST110_BACKUP_RUNTIME_TEST_RECEIPT_FAULT' in source
|
||||
assert 'HOST110_BACKUP_RUNTIME_TEST_APPLY_FAULT' in source
|
||||
assert 'HOST110_BACKUP_RUNTIME_TEST_ROLLBACK_FAULT' in source
|
||||
assert 'HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL' in source
|
||||
assert 'HOST110_BACKUP_RUNTIME_TEST_TERMINAL_SIGNAL' in source
|
||||
assert 'HOST110_BACKUP_RUNTIME_TEST_TERMINAL_RECEIPT_FAULT' in source
|
||||
assert 'HOST110_BACKUP_RUNTIME_TEST_CLEANUP_FAULT' in source
|
||||
assert 'HOST110_BACKUP_RUNTIME_ENABLE_TEST_FAULTS' in source
|
||||
assert '[ "$DEST_ROOT" != "/backup/scripts" ]' in source
|
||||
assert 'fsync_rollback_prestate || return 75' in source
|
||||
@@ -401,8 +794,18 @@ def test_host110_executor_is_manifest_bound_atomic_and_fail_closed() -> None:
|
||||
assert 'os.fsync(handle.fileno())' in source
|
||||
assert 'os.fsync(directory_fd)' in source
|
||||
assert 'hashlib.sha256(readback).hexdigest()' in source
|
||||
assert source.index('write_receipt "verified"') < source.index("\nAPPLY_COMPLETE=1")
|
||||
assert source.index("\nAPPLY_COMPLETE=1") < source.rindex('rm -rf -- "$ROLLBACK_DIR"')
|
||||
assert source.index("COMMIT_CRITICAL=1") < source.index('write_receipt "payload_verified"')
|
||||
assert source.index('write_receipt "payload_verified"') < source.index("\nAPPLY_COMPLETE=1")
|
||||
assert source.index("\nAPPLY_COMPLETE=1") < source.rindex("\nCOMMIT_CRITICAL=0")
|
||||
assert source.index("\nAPPLY_COMPLETE=1") < source.rindex(
|
||||
'cleanup_postcommit_directory "rollback_prestate"'
|
||||
)
|
||||
assert 'handle_agent99_signal HUP 129' in source
|
||||
assert 'handle_agent99_signal TERM 143' in source
|
||||
assert source.index("trap '' HUP INT TERM") < source.index('write_terminal_receipt "$terminal_status"')
|
||||
assert 'write_terminal_receipt "$terminal_status"' in source
|
||||
assert 'terminal_status="cleanup_pending"' in source
|
||||
assert 'terminal_status="committed_signal_deferred"' in source
|
||||
assert 'run_identity_receipt_exists' in source
|
||||
assert 'run_identity_stage_exists' in source
|
||||
assert 'run_identity_rollback_exists' in source
|
||||
@@ -432,22 +835,205 @@ def test_apply_success_publishes_durable_receipt_before_prestate_cleanup(tmp_pat
|
||||
assert output["zeroResidueVerified"] is True
|
||||
assert output["stageCleanupVerified"] is True
|
||||
assert output["rollbackPrestateCleanupVerified"] is True
|
||||
assert output["status"] == "verified"
|
||||
assert output["payloadCommitted"] is True
|
||||
assert output["runId"] == run_id
|
||||
assert output["terminalExitCode"] == 0
|
||||
assert output["terminalReceiptPublished"] is True
|
||||
assert output["deferredSignal"] is None
|
||||
receipt_path = Path(harness["status_root"]) / f"agent99-backup-runtime-deploy-{run_id}.json"
|
||||
receipt = json.loads(receipt_path.read_text(encoding="utf-8"))
|
||||
assert receipt["status"] == "verified"
|
||||
assert receipt["status"] == "payload_verified"
|
||||
assert receipt["rollbackAttempted"] is False
|
||||
assert receipt["rollbackPerformed"] is False
|
||||
assert receipt["rollbackVerified"] is False
|
||||
assert receipt["zeroResidueVerified"] is True
|
||||
assert receipt["zeroResidueScope"] == "run_owned_destination_temps"
|
||||
terminal_path = Path(harness["status_root"]) / f"agent99-backup-runtime-terminal-{run_id}.json"
|
||||
terminal = json.loads(terminal_path.read_text(encoding="utf-8"))
|
||||
assert terminal["status"] == "verified"
|
||||
assert terminal["ok"] is True
|
||||
assert terminal["payloadCommitted"] is True
|
||||
assert terminal["stageCleanupVerified"] is True
|
||||
assert terminal["rollbackPrestateCleanupVerified"] is True
|
||||
assert terminal["terminalExitCode"] == 0
|
||||
assert terminal["zeroResidueScope"] == "run_owned_destination_temps_and_internal_stage_prestate"
|
||||
assert output["terminalReceipt"] == str(terminal_path)
|
||||
assert not (Path(harness["rollback_root"]) / run_id).exists()
|
||||
assert not (Path(harness["stage_root"]) / run_id).exists()
|
||||
for name in harness["payload_order"]:
|
||||
target = (
|
||||
Path(harness["exporter_root"]) / name
|
||||
if name == "backup-health-textfile-exporter.py"
|
||||
else Path(harness["destination"]) / name
|
||||
)
|
||||
assert target.read_bytes() == (Path(harness["source_stage"]) / name).read_bytes()
|
||||
_assert_payload_committed(harness)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("signal_name", "exit_code"), [("TERM", 143), ("HUP", 129), ("INT", 130)])
|
||||
def test_commit_window_signal_is_deferred_until_payload_and_terminal_are_consistent(
|
||||
tmp_path: Path,
|
||||
signal_name: str,
|
||||
exit_code: int,
|
||||
) -> None:
|
||||
run_id = f"commit-signal-{signal_name.lower()}"
|
||||
harness = _build_executor_harness(tmp_path, run_id)
|
||||
|
||||
result = _run_executor_harness(
|
||||
harness,
|
||||
HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL=signal_name,
|
||||
)
|
||||
|
||||
assert result.returncode == exit_code, result.stderr
|
||||
output = json.loads(result.stdout)
|
||||
assert output["ok"] is False
|
||||
assert output["status"] == "committed_signal_deferred"
|
||||
assert output["payloadCommitted"] is True
|
||||
assert output["runId"] == run_id
|
||||
assert output["deferredSignal"] == signal_name
|
||||
assert output["terminalExitCode"] == exit_code
|
||||
assert output["terminalReceiptPublished"] is True
|
||||
assert output["stageCleanupVerified"] is True
|
||||
assert output["rollbackPrestateCleanupVerified"] is True
|
||||
_assert_payload_committed(harness)
|
||||
payload_receipt = json.loads(
|
||||
(
|
||||
Path(harness["status_root"])
|
||||
/ f"agent99-backup-runtime-deploy-{run_id}.json"
|
||||
).read_text(encoding="utf-8")
|
||||
)
|
||||
assert payload_receipt["status"] == "payload_verified"
|
||||
terminal = json.loads(
|
||||
(
|
||||
Path(harness["status_root"])
|
||||
/ f"agent99-backup-runtime-terminal-{run_id}.json"
|
||||
).read_text(encoding="utf-8")
|
||||
)
|
||||
assert terminal["status"] == "committed_signal_deferred"
|
||||
assert terminal["ok"] is False
|
||||
assert terminal["deferredSignal"] == signal_name
|
||||
assert terminal["terminalExitCode"] == exit_code
|
||||
assert not (Path(harness["rollback_root"]) / run_id).exists()
|
||||
assert not (Path(harness["stage_root"]) / run_id).exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("signal_name", ["TERM", "HUP", "INT"])
|
||||
def test_terminal_publication_masks_late_signal_without_receipt_stdout_exit_divergence(
|
||||
tmp_path: Path,
|
||||
signal_name: str,
|
||||
) -> None:
|
||||
run_id = f"terminal-signal-{signal_name.lower()}"
|
||||
harness = _build_executor_harness(tmp_path, run_id)
|
||||
|
||||
result = _run_executor_harness(
|
||||
harness,
|
||||
HOST110_BACKUP_RUNTIME_TEST_TERMINAL_SIGNAL=signal_name,
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
output = json.loads(result.stdout)
|
||||
assert output["ok"] is True
|
||||
assert output["status"] == "verified"
|
||||
assert output["payloadCommitted"] is True
|
||||
assert output["runId"] == run_id
|
||||
assert output["deferredSignal"] is None
|
||||
assert output["terminalExitCode"] == 0
|
||||
assert output["terminalReceiptPublished"] is True
|
||||
terminal = json.loads(
|
||||
(
|
||||
Path(harness["status_root"])
|
||||
/ f"agent99-backup-runtime-terminal-{run_id}.json"
|
||||
).read_text(encoding="utf-8")
|
||||
)
|
||||
assert terminal["ok"] is True
|
||||
assert terminal["status"] == output["status"]
|
||||
assert terminal["deferredSignal"] == output["deferredSignal"]
|
||||
assert terminal["terminalExitCode"] == output["terminalExitCode"]
|
||||
_assert_payload_committed(harness)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fault", ["write", "link", "fsync", "readback"])
|
||||
def test_terminal_receipt_fault_is_nonzero_with_payload_receipt_only(
|
||||
tmp_path: Path,
|
||||
fault: str,
|
||||
) -> None:
|
||||
run_id = f"terminal-receipt-{fault}"
|
||||
harness = _build_executor_harness(tmp_path, run_id)
|
||||
|
||||
result = _run_executor_harness(
|
||||
harness,
|
||||
HOST110_BACKUP_RUNTIME_TEST_TERMINAL_RECEIPT_FAULT=fault,
|
||||
)
|
||||
|
||||
assert result.returncode == 76
|
||||
assert '"ok": true' not in result.stdout.lower()
|
||||
assert "terminal_receipt_failed" in result.stderr
|
||||
output = json.loads(result.stdout)
|
||||
assert output["ok"] is False
|
||||
assert output["status"] == "terminal_receipt_failed"
|
||||
assert output["payloadCommitted"] is True
|
||||
assert output["runId"] == run_id
|
||||
assert output["terminalReceiptPublished"] is False
|
||||
assert output["terminalExitCode"] == 76
|
||||
_assert_payload_committed(harness)
|
||||
payload_receipt = json.loads(
|
||||
(
|
||||
Path(harness["status_root"])
|
||||
/ f"agent99-backup-runtime-deploy-{run_id}.json"
|
||||
).read_text(encoding="utf-8")
|
||||
)
|
||||
assert payload_receipt["status"] == "payload_verified"
|
||||
assert not (
|
||||
Path(harness["status_root"])
|
||||
/ f"agent99-backup-runtime-terminal-{run_id}.json"
|
||||
).exists()
|
||||
assert not (Path(harness["rollback_root"]) / run_id).exists()
|
||||
assert not (Path(harness["stage_root"]) / run_id).exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("fault", "stage_clean", "rollback_clean"),
|
||||
[("stage", False, True), ("rollback_prestate", True, False)],
|
||||
)
|
||||
def test_postcommit_cleanup_failure_is_nonzero_and_never_false_green(
|
||||
tmp_path: Path,
|
||||
fault: str,
|
||||
stage_clean: bool,
|
||||
rollback_clean: bool,
|
||||
) -> None:
|
||||
run_id = f"cleanup-{fault}"
|
||||
harness = _build_executor_harness(tmp_path, run_id)
|
||||
|
||||
result = _run_executor_harness(
|
||||
harness,
|
||||
HOST110_BACKUP_RUNTIME_TEST_CLEANUP_FAULT=fault,
|
||||
)
|
||||
|
||||
assert result.returncode == 75, result.stderr
|
||||
assert '"ok": true' not in result.stdout.lower()
|
||||
output = json.loads(result.stdout)
|
||||
assert output["ok"] is False
|
||||
assert output["status"] == "cleanup_pending"
|
||||
assert output["payloadCommitted"] is True
|
||||
assert output["runId"] == run_id
|
||||
assert output["stageCleanupVerified"] is stage_clean
|
||||
assert output["rollbackPrestateCleanupVerified"] is rollback_clean
|
||||
assert output["terminalReceiptPublished"] is True
|
||||
_assert_payload_committed(harness)
|
||||
payload_receipt = json.loads(
|
||||
(
|
||||
Path(harness["status_root"])
|
||||
/ f"agent99-backup-runtime-deploy-{run_id}.json"
|
||||
).read_text(encoding="utf-8")
|
||||
)
|
||||
assert payload_receipt["status"] == "payload_verified"
|
||||
terminal = json.loads(
|
||||
(
|
||||
Path(harness["status_root"])
|
||||
/ f"agent99-backup-runtime-terminal-{run_id}.json"
|
||||
).read_text(encoding="utf-8")
|
||||
)
|
||||
assert terminal["status"] == "cleanup_pending"
|
||||
assert terminal["ok"] is False
|
||||
assert terminal["payloadCommitted"] is True
|
||||
assert terminal["stageCleanupVerified"] is stage_clean
|
||||
assert terminal["rollbackPrestateCleanupVerified"] is rollback_clean
|
||||
assert (Path(harness["stage_root"]) / run_id).exists() is (not stage_clean)
|
||||
assert (Path(harness["rollback_root"]) / run_id).exists() is (not rollback_clean)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("fault", ["write", "link", "fsync", "readback"])
|
||||
@@ -539,8 +1125,258 @@ def test_backup_runtime_entrypoints_share_the_deployment_gate() -> None:
|
||||
for source in (common, restore, offsite):
|
||||
assert "/tmp/agent99-host110-backup-runtime.lock" in source
|
||||
assert "flock -s -w 60" in source
|
||||
assert "realpath -e" in source
|
||||
assert "readlink -f" in source
|
||||
assert "identity unresolved" in source
|
||||
assert "/proc/$$/fd/197" in source
|
||||
assert "/dev/fd/197" in source
|
||||
assert "inherited fd 197 identity mismatch" in source
|
||||
assert source.index("backup_runtime_inherited_fd_status") < source.index("exec 197>>")
|
||||
assert 'resolve_backup_runtime_source_path "${BASH_SOURCE[1]:-${BASH_SOURCE[0]:-}}"' in common
|
||||
assert '${PWD}/${raw_path}' in common
|
||||
assert '[ "${BACKUP_RUNTIME_SHARED_LOCK_HELD:-0}" != "1" ] || return 0' not in common
|
||||
for source in (restore, offsite):
|
||||
assert 'resolve_backup_runtime_script_path "${BASH_SOURCE[0]:-}"' in source
|
||||
assert '&& [ "${BACKUP_RUNTIME_SHARED_LOCK_HELD:-0}" != "1" ]' not in source
|
||||
assert "fcntl.LOCK_SH | fcntl.LOCK_NB" in archive
|
||||
assert "BACKUP_RUNTIME_SHARED_LOCK_HELD" in archive
|
||||
assert "BACKUP_RUNTIME_SHARED_LOCK_HELD" not in archive
|
||||
assert 'Path(f"/proc/{os.getpid()}/fd")' in archive
|
||||
assert 'Path("/dev/fd/197")' in archive
|
||||
assert "os.dup(197)" in archive
|
||||
|
||||
|
||||
def test_common_shared_lock_preserves_resolved_nonproduction_fixture_behavior(tmp_path: Path) -> None:
|
||||
gate_source = (ROOT / "scripts/backup/common.sh").read_text(encoding="utf-8").split(
|
||||
"# WOOO AIOps - 備份共用函式庫",
|
||||
1,
|
||||
)[0]
|
||||
gate_fixture = tmp_path / "common-gate.sh"
|
||||
_write_executable(gate_fixture, gate_source)
|
||||
wrapper = tmp_path / "fixture-entrypoint.sh"
|
||||
_write_executable(
|
||||
wrapper,
|
||||
"#!/usr/bin/env bash\n"
|
||||
"set -euo pipefail\n"
|
||||
f"source {str(gate_fixture)!r}\n"
|
||||
"printf 'NONPRODUCTION_FIXTURE_OK=1\\n'\n",
|
||||
)
|
||||
|
||||
result = subprocess.run(
|
||||
[str(wrapper)],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "NONPRODUCTION_FIXTURE_OK=1" in result.stdout
|
||||
|
||||
|
||||
@pytest.mark.parametrize("entrypoint", ["common", "restore", "offsite"])
|
||||
@pytest.mark.parametrize("invocation", ["absolute", "relative", "symlink"])
|
||||
def test_backup_runtime_shared_lock_blocks_absolute_relative_and_symlink_invocation(
|
||||
tmp_path: Path,
|
||||
entrypoint: str,
|
||||
invocation: str,
|
||||
) -> None:
|
||||
script_path, runtime_root, lock_path, environment = _build_shared_lock_gate_fixture(
|
||||
tmp_path,
|
||||
entrypoint,
|
||||
)
|
||||
|
||||
command_path = str(script_path)
|
||||
cwd = runtime_root
|
||||
if invocation == "relative":
|
||||
command_path = f"./{script_path.name}"
|
||||
elif invocation == "symlink":
|
||||
link_path = tmp_path / f"{entrypoint}-entrypoint-link"
|
||||
link_path.symlink_to(script_path)
|
||||
command_path = str(link_path)
|
||||
|
||||
environment["BACKUP_RUNTIME_SHARED_LOCK_HELD"] = "1"
|
||||
lock_handle = lock_path.open("a+")
|
||||
fcntl.flock(lock_handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[str(command_path)],
|
||||
cwd=cwd,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
timeout=3,
|
||||
)
|
||||
finally:
|
||||
fcntl.flock(lock_handle.fileno(), fcntl.LOCK_UN)
|
||||
lock_handle.close()
|
||||
|
||||
assert result.returncode == 75
|
||||
assert "backup runtime deployment is active" in result.stderr
|
||||
assert "BODY_REACHED=1" not in result.stdout
|
||||
|
||||
|
||||
@pytest.mark.parametrize("entrypoint", ["common", "restore", "offsite"])
|
||||
def test_backup_runtime_shared_lock_reuses_exact_inherited_fd_without_reopen(
|
||||
tmp_path: Path,
|
||||
entrypoint: str,
|
||||
) -> None:
|
||||
script_path, runtime_root, lock_path, environment = _build_shared_lock_gate_fixture(
|
||||
tmp_path,
|
||||
entrypoint,
|
||||
reject_fd_reopen=True,
|
||||
body_sleep_seconds=0.5,
|
||||
)
|
||||
wrapper = tmp_path / "run-with-inherited-lock.sh"
|
||||
_write_executable(
|
||||
wrapper,
|
||||
"#!/usr/bin/env bash\n"
|
||||
"set -euo pipefail\n"
|
||||
f"exec 197>>{str(lock_path)!r}\n"
|
||||
"flock -s -w 1 197\n"
|
||||
"export BACKUP_RUNTIME_SHARED_LOCK_HELD=forged_external_value\n"
|
||||
f"exec {str(script_path)!r}\n",
|
||||
)
|
||||
|
||||
process = subprocess.Popen(
|
||||
[str(wrapper)],
|
||||
cwd=runtime_root,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
text=True,
|
||||
env=environment,
|
||||
)
|
||||
assert process.stdout is not None
|
||||
assert process.stdout.readline().strip() == "BODY_REACHED=1"
|
||||
contender = lock_path.open("a+")
|
||||
try:
|
||||
with pytest.raises(BlockingIOError):
|
||||
fcntl.flock(contender.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
finally:
|
||||
contender.close()
|
||||
stdout, stderr = process.communicate(timeout=3)
|
||||
|
||||
assert process.returncode == 0, stderr
|
||||
assert "FD_REOPENED=1" not in stderr
|
||||
assert stdout == ""
|
||||
|
||||
|
||||
@pytest.mark.parametrize("entrypoint", ["common", "restore", "offsite"])
|
||||
def test_backup_runtime_shared_lock_rejects_inherited_fd_for_another_file(
|
||||
tmp_path: Path,
|
||||
entrypoint: str,
|
||||
) -> None:
|
||||
script_path, runtime_root, _, environment = _build_shared_lock_gate_fixture(tmp_path, entrypoint)
|
||||
wrong_lock = tmp_path / "wrong-runtime.lock"
|
||||
wrapper = tmp_path / "run-with-wrong-inherited-lock.sh"
|
||||
_write_executable(
|
||||
wrapper,
|
||||
"#!/usr/bin/env bash\n"
|
||||
"set -euo pipefail\n"
|
||||
f"exec 197>>{str(wrong_lock)!r}\n"
|
||||
"export BACKUP_RUNTIME_SHARED_LOCK_HELD=1\n"
|
||||
f"exec {str(script_path)!r}\n",
|
||||
)
|
||||
|
||||
result = subprocess.run(
|
||||
[str(wrapper)],
|
||||
cwd=runtime_root,
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
timeout=3,
|
||||
)
|
||||
|
||||
assert result.returncode == 69
|
||||
assert "inherited fd 197 identity mismatch" in result.stderr
|
||||
assert "BODY_REACHED=1" not in result.stdout
|
||||
|
||||
|
||||
def test_host188_archive_lock_ignores_forged_env_and_validates_inherited_fd(tmp_path: Path) -> None:
|
||||
runtime_root = tmp_path / "runtime-scripts"
|
||||
runtime_root.mkdir()
|
||||
lock_path = tmp_path / "runtime.lock"
|
||||
archive_path = runtime_root / "verify-host188-products-archive.py"
|
||||
archive_source = (ROOT / "scripts/backup/verify-host188-products-archive.py").read_text(encoding="utf-8")
|
||||
archive_source = archive_source.replace('Path("/backup/scripts")', f"Path({str(runtime_root)!r})")
|
||||
archive_source = archive_source.replace(
|
||||
'RUNTIME_LOCK = Path("/tmp/agent99-host110-backup-runtime.lock")',
|
||||
f"RUNTIME_LOCK = Path({str(lock_path)!r})",
|
||||
)
|
||||
archive_source = archive_source.replace("time.monotonic() + 60", "time.monotonic() + 0.2")
|
||||
_write_executable(archive_path, archive_source)
|
||||
runner = tmp_path / "archive-lock-replay.py"
|
||||
runner.write_text(
|
||||
"""import fcntl
|
||||
import importlib.util
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
archive_path = pathlib.Path(sys.argv[1])
|
||||
lock_path = pathlib.Path(sys.argv[2])
|
||||
mode = sys.argv[3]
|
||||
spec = importlib.util.spec_from_file_location("host188_archive_lock_replay", archive_path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
if mode in {"exact", "wrong"}:
|
||||
selected = lock_path if mode == "exact" else lock_path.with_name("wrong-runtime.lock")
|
||||
selected.touch()
|
||||
inherited = selected.open("a+")
|
||||
fcntl.flock(inherited.fileno(), fcntl.LOCK_SH | fcntl.LOCK_NB)
|
||||
os.dup2(inherited.fileno(), 197)
|
||||
try:
|
||||
handle = module.acquire_runtime_lock()
|
||||
except RuntimeError as exc:
|
||||
print(str(exc))
|
||||
raise SystemExit(69)
|
||||
else:
|
||||
print("LOCK_OK=1")
|
||||
if handle is not None:
|
||||
handle.close()
|
||||
""",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
environment = os.environ.copy()
|
||||
environment["BACKUP_RUNTIME_SHARED_LOCK_HELD"] = "1"
|
||||
lock_handle = lock_path.open("a+")
|
||||
fcntl.flock(lock_handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
try:
|
||||
forged = subprocess.run(
|
||||
[sys.executable, str(runner), str(archive_path), str(lock_path), "forged"],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
timeout=3,
|
||||
)
|
||||
finally:
|
||||
fcntl.flock(lock_handle.fileno(), fcntl.LOCK_UN)
|
||||
lock_handle.close()
|
||||
exact = subprocess.run(
|
||||
[sys.executable, str(runner), str(archive_path), str(lock_path), "exact"],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
timeout=3,
|
||||
)
|
||||
wrong = subprocess.run(
|
||||
[sys.executable, str(runner), str(archive_path), str(lock_path), "wrong"],
|
||||
check=False,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=environment,
|
||||
timeout=3,
|
||||
)
|
||||
|
||||
assert forged.returncode == 69
|
||||
assert "backup_runtime_deployment_active" in forged.stdout
|
||||
assert exact.returncode == 0, exact.stdout + exact.stderr
|
||||
assert "LOCK_OK=1" in exact.stdout
|
||||
assert wrong.returncode == 69
|
||||
assert "backup_runtime_inherited_fd_mismatch" in wrong.stdout
|
||||
|
||||
|
||||
def test_independent_verifier_accepts_exact_bundle_without_writing(tmp_path, monkeypatch, capsys) -> None:
|
||||
|
||||
Reference in New Issue
Block a user