Compare commits

...

4 Commits

Author SHA1 Message Date
Your Name
2c2dc086b0 fix(agent99): fail closed host110 runtime replay 2026-07-19 02:40:26 +08:00
Your Name
d1797a1c87 fix(agent99): close host110 backup commit races 2026-07-19 01:08:45 +08:00
Your Name
8f11ef3362 fix(backup): make Host110 runtime receipt durable 2026-07-18 23:37:52 +08:00
Your Name
0b9c028400 fix(backup): align Host110 runtime manifest staging 2026-07-18 22:38:53 +08:00
12 changed files with 3998 additions and 77 deletions

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,217 @@
# AWOOOI Agent99 Host110 Backup Runtime V10
Status: proposed; central review and explicit owner approval are required before
any production apply.
This artifact supersedes V2 through V9. Every V9 identity below is void for
approval and remains audit evidence only:
- branch/ref: `codex/host110-backup-runtime-v9-20260718`
- candidate/revision: `8f11ef3362a8d7f66a46308b58fd142b0c91a8bc`
- artifact SHA-256: `929978eb3da15b04791a8614e6d5ee504ae2a4c2f205b694307a7bad7713bcad`
- exact diff SHA-256: `8085d49450b74e7c43577ba118cbd74d3a2e3dd2d22eeba90ce749d56c72c0f2`
- every other V9 proposal, content, revision, and derived hash
No V9 hash, ref, review, receipt, or test grants production execution
authority.
## Exact Scope
- Agent99 runtime bundle: 19 files, guarded by one producer/consumer parity
test.
- Host110 backup payload: 18 ordered files, comprising 17 backup scripts and
`backup-health-textfile-exporter.py`.
- Host110 remote stage: 21 unique basenames, comprising the ordered 18-file
payload, fixed executor, independent verifier, and `manifest.json`.
- Fixed target: `wooo@192.168.0.110`, dispatched by Windows99 Agent99 from an
exact Gitea revision and digest manifest.
- The candidate includes a 10-line runtime-lock addition to
`scripts/backup/gitea-full-backup-restore-drill.sh`. A future drill invocation
acquires the shared Host110 backup-runtime lock before its pre-existing drill
behavior. Applying this candidate replaces the script but does not invoke the
drill or any container lifecycle.
- `scripts/backup/backup-gitea.sh` has zero diff from the production base. No
Gitea primary stop/start or backup-container creation is part of this
candidate apply.
- V7, V8, and V9 proposal documents remain tracked audit artifacts and are not
runtime authority.
- `agent99-host110-broker-contract-replay.ps1` is a no-write Windows99 test
asset; it is not part of the 19-file Agent99 runtime or 18-file Host110
payload.
## V9 Defects Closed
### Signal-Safe Commit Boundary
The executor enters a signal-deferring critical section before publishing the
payload receipt. `INT`, `HUP`, and `TERM` are recorded without exiting until the
durable payload-receipt readback and `APPLY_COMPLETE=1` commit state are both
established. A deferred signal can never enter the pre-commit EXIT rollback
path after the payload receipt exists.
After commit, exact stage and rollback-prestate cleanup still run. If cleanup
is complete, a deferred signal produces the non-success terminal
`committed_signal_deferred`, a durable terminal receipt, the corresponding
129/130/143 exit, and an applied payload. TERM, HUP, and INT are replayed in the
exact receipt-to-commit window.
After cleanup, terminal publication uses a bounded signal mask. Signals captured
before that boundary remain part of the non-success terminal; signals delivered
during the immutable terminal writer cannot mutate its already-final state.
Terminal-writer TERM, HUP, and INT replays require receipt, stdout, payload, and
exit to remain consistent.
### Payload And Terminal Receipts
The immutable payload receipt has status `payload_verified`; it proves exact
payload, independent verifier, durable receipt readback, and absence of
run-owned destination temporaries. It does not claim internal stage or rollback
prestate cleanup.
A separate immutable terminal receipt is the authority for overall success.
Only status `verified` with `stageCleanupVerified=true`,
`rollbackPrestateCleanupVerified=true`, no deferred signal, and exit code zero
may yield executor `ok:true`. Each successful cleanup is absence-read back and
its parent directory is fsynced before terminal publication.
If either cleanup fails, the payload remains committed, exact residue is
preserved, terminal status is `cleanup_pending`, executor output is
`ok:false`, and exit is nonzero. A payload receipt cannot be used as a
zero-residue or broker-success shortcut. The Windows99 broker validates every
status, payload, rollback, residue-scope, cleanup, signal, exit, receipt-path,
and independent-verifier field fail-closed.
Committed non-success outputs are parsed and retained in broker evidence before
the broker fails overall; `cleanup_pending`, `committed_signal_deferred`, and
`terminal_receipt_failed` can never be collapsed into an ambiguous generic
transport failure.
### Existing Transaction Guarantees Retained
- Producer and executor payload basenames are exact-equal in count, set, and
order: 18 payload files and 21 unique staged files.
- Receipt write, hard-link, fsync, and readback failures roll back exact content
and metadata without publishing success.
- Terminal-receipt write, hard-link, fsync, and readback failures keep the
verified payload, publish no terminal success, emit structured committed
failure evidence with `terminalReceiptPublished=false`, and exit nonzero.
- Apply and rollback destination temporaries are run-owned, tracked, removed,
and absence-read back.
- Rollback attempted, performed, verified, and residue truth remain separate;
partial or unknown rollback terminates `rollback_unverified`.
- Fault hooks require both the explicit test flag and a transformed
non-production destination; canonical `/backup/scripts` cannot enable them.
## Apply Effects
- Windows99 Agent99 runtime promotion writes the 19 runtime files plus bounded
manifest and receipt state.
- Host110 Apply replaces the ordered 18 payload paths through one exclusive,
digest-bound filesystem transaction, and writes payload/terminal receipts
below `/backup/status`.
- The deployed restore-drill script gains shared-lock behavior. The apply does
not execute that script.
- Apply does not run a backup, invoke a Gitea container lifecycle, sync Google
Drive, prune snapshots, restart a VM or service, or change a database,
network, firewall, credential, or secret.
## Reproducible Validation
Focused replay and contract suite:
```sh
pytest -q \
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
scripts/backup/tests/test_verify_offsite_full_sync_runtime.py \
scripts/ops/tests/test_agent99_signoz_metadata_executor_runtime_entrypoint.py \
scripts/ops/tests/test_db_bounded_executor_runtime_entrypoint.py \
scripts/reboot-recovery/tests/test_agent99_live_preflight_decision.py \
scripts/reboot-recovery/tests/test_agent99_remote_atomic_deploy_transport_contract.py \
scripts/reboot-recovery/tests/test_host112_manager_recovery_contract.py
```
Exact signal and post-commit cleanup replays:
```sh
pytest -vv \
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
-k 'commit_window_signal or terminal_publication or terminal_receipt_fault or postcommit_cleanup'
```
Static, syntax, and repository checks:
```sh
ruff check scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py
bash -n scripts/backup/host110-backup-runtime-executor.sh
git diff --check
PATH=/Users/ogt/.pyenv/shims:$PATH bash scripts/ops/ansible-validate.sh
```
`ansible-validate.sh` parses
`ops/reboot-recovery/full-stack-backup-baseline.yml` as YAML metadata. It runs
`ansible-playbook --syntax-check` only for the actual playbooks listed by the
script; the metadata mapping is not represented as a playbook.
Windows99 no-file-write PowerShell parser check:
```sh
PARSER_COMMAND='$source=[Console]::In.ReadToEnd(); $tokens=$null; $errors=$null; [System.Management.Automation.Language.Parser]::ParseInput($source,[ref]$tokens,[ref]$errors) | Out-Null; [pscustomobject]@{ok=($errors.Count -eq 0); errorCount=$errors.Count; parser="WindowsPowerShell"} | ConvertTo-Json -Compress; if ($errors.Count -ne 0) { exit 1 }'
ENCODED=$(printf '%s' "$PARSER_COMMAND" | iconv -f UTF-8 -t UTF-16LE | base64 | tr -d '\n')
ssh -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=5 \
-o ConnectionAttempts=1 Administrator@192.168.0.99 \
"powershell.exe -NoProfile -NonInteractive -EncodedCommand $ENCODED" \
< agent99-host110-backup-runtime-broker.ps1
```
Windows99 actual-function missing-field/type contract replay:
```sh
WRAPPER='$script=[Console]::In.ReadToEnd(); & ([ScriptBlock]::Create($script))'
WRAPPER_ENCODED=$(printf '%s' "$WRAPPER" | iconv -f UTF-8 -t UTF-16LE | base64 | tr -d '\n')
BROKER_GZIP_BASE64=$(gzip -c agent99-host110-backup-runtime-broker.ps1 | base64 | tr -d '\n')
awk -v payload="$BROKER_GZIP_BASE64" \
'BEGIN { print "$BrokerGzipBase64 = \"" payload "\"" } { print }' \
scripts/reboot-recovery/tests/agent99-host110-broker-contract-replay.ps1 |
ssh -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=5 \
-o ConnectionAttempts=1 Administrator@192.168.0.99 \
"powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass -EncodedCommand $WRAPPER_ENCODED"
```
The replay loads the candidate broker's actual function AST, accepts the valid
success and three committed non-success terminals, then removes every required
field and injects wrong JSON types. Expected readback is 34 missing-field and 6
type rejections. It performs no remote file write.
## Exact Diff Serialization
Substitute the V10 Gitea live-ref SHA for `<candidate_git_sha>` and hash the
exact stdout bytes:
```sh
git diff --binary --full-index \
e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3..<candidate_git_sha> |
shasum -a 256
```
Central review must independently bind the live feature ref and candidate Git
SHA, this tracked artifact path and bytes hash, the serialized diff hash/path
count/line counts, the 19/18/21 count contracts, and the exact restore-drill and
`backup-gitea.sh` diffs.
## Rollback
- Source rollback reverts the exact V10 diff to
`e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3` without force-pushing.
- Agent99 rollback restores the prior runtime files and manifest and verifies
prior hashes independently.
- Before payload commit, Host110 rollback restores all prior payload content,
SHA-256, uid, gid, and mode under the exclusive lock and proves no run-owned
destination temporary remains.
- After payload commit, the executor never rolls the payload back merely due to
a signal or cleanup error. It preserves exact cleanup evidence and reports a
non-success terminal. Any later cleanup-only remediation is a separate
controlled action scoped to the run-specific stage/prestate paths and may not
rewrite the verified payload.
- Source rollback restores the restore-drill script's prior unlocked behavior.
It does not invoke the drill or any container lifecycle.
- Any canonical, metadata, receipt, signal, cleanup, or residue mismatch fails
closed and cannot produce broker `verified`.

View File

@@ -0,0 +1,202 @@
# AWOOOI Agent99 Host110 Backup Runtime V11
Status: proposed. Central review and explicit owner approval are required before
any production apply. This candidate must not be pushed to `main`, sent through
CD, or applied to Windows99 or Host110 while it remains proposed.
This artifact supersedes V10. Every V10 identity below is void for approval and
remains audit evidence only:
- branch/ref: `codex/host110-backup-runtime-v10-20260718`
- candidate/revision: `d1797a1c8766722ae00ac14e20fed00f4e3f3a5b`
- artifact SHA-256: `eb48661bbecc4e169111879bb18dfab63be29878d5f5c3651d9917f81f195d28`
- exact diff SHA-256: `f7efb98d5f91fa6899713b692c8bb015fcb6ae6dba4291cc6ae429290072964e`
- every other V10 proposal, content, revision, review, and derived hash
No V10 hash, ref, review, receipt, or test grants production execution
authority.
## Exact Scope
- Agent99 runtime bundle: 19 files, guarded by one producer/consumer parity
test.
- Host110 backup payload: 18 ordered files, comprising 17 backup scripts and
`backup-health-textfile-exporter.py`.
- Host110 remote stage: 21 unique basenames, comprising the ordered 18-file
payload, fixed executor, independent verifier, and `manifest.json`.
- Fixed target: `wooo@192.168.0.110`, dispatched only by the Windows99 Agent99
typed broker from an exact Gitea revision and digest-bound manifest.
- V11 changes the Windows99 broker, three shell shared-lock entrypoints, the
Host188 archive verifier's shared-lock acquisition, and their no-write replay
tests.
- `scripts/backup/backup-gitea.sh` has zero diff from the production base. No
Gitea primary stop/start or backup-container creation is part of this apply.
- The restore drill and offsite gate gain only canonical shared-lock behavior.
Applying this candidate replaces those scripts but does not invoke a drill,
run a backup, or start an offsite sync.
- Prior proposal documents remain tracked audit artifacts and are not runtime
authority.
## V10 Defects Closed
### Persisted Broker Evidence Is Exact And Fail-Closed
Before reusing a run-bound broker receipt, the broker now validates exact key
sets, required native JSON types, schema version, run ID, source revision, mode,
target, evidence path binding, status, error, cleanup, and nested Apply/Verify
contracts. Missing, extra, wrong-type, wrong-mode, wrong-run, conflicting, or
malformed evidence fails closed before any replay decision.
A non-empty string such as `"false"` can no longer be cast to a truthy Boolean.
An existing failure remains a failure and cannot be upgraded to success merely
because it is parseable or shares a source revision.
### Mandatory Post-Commit Transport Reconciliation
If Apply loses SSH transport or times out, the broker performs a read-only,
exact-RunId receipt readback under `/backup/status` and an independent
destination verifier readback. It never blindly retries Apply and never
collapses a post-commit state into a generic precommit failure.
The reconciliation states are explicit and non-success:
- `committed_verified_terminal_missing`
- `committed_verified_transport_lost`
- `committed_cleanup_pending_transport_lost`
- `committed_signal_deferred_transport_lost`
- `committed_unknown`
Every state records `retryApplyAllowed=false`. Receipt schema, native types,
source/run/head identity, payload paths and counts, verifier identity, cleanup,
signal, and terminal semantics are validated before classification. The
readback performs no remote write and reads no secret value.
### Canonical Shared Lock And Nested Re-entry
Production backup entrypoints canonicalize absolute, relative, and symlink
invocation paths before deciding whether the Host110 runtime lock applies.
Unresolvable production identity fails closed.
For nested calls, an inherited FD 197 is accepted only when its canonical
target is the exact runtime lock. Linux uses `/proc/$$/fd/197`; the bounded
non-Linux fallback resolves the descriptor path with `F_GETPATH`. The entrypoint
then requests the shared lock on the same descriptor. It does not close and
reopen a valid inherited descriptor, so there is no release/reacquire gap.
If FD 197 is absent, the entrypoint opens the lock and acquires the shared side.
If FD 197 exists but points elsewhere or cannot be resolved, execution fails
closed. `BACKUP_RUNTIME_SHARED_LOCK_HELD` and all other external environment
values grant no lock authority.
The Host188 archive verifier follows the same contract: exact inherited FD 197
is duplicated from the same open file description, wrong identity fails
closed, and a forged environment value cannot bypass an active exclusive lock.
## Apply Effects
- Windows99 Agent99 runtime promotion writes the 19 runtime files plus bounded
manifest and receipt state.
- Host110 Apply replaces the ordered 18 payload paths through one exclusive,
digest-bound filesystem transaction and writes payload/terminal receipts
below `/backup/status`.
- The deployed common library, restore drill, offsite gate, and Host188 archive
verifier gain the shared-lock behavior described above. Apply does not invoke
their backup, drill, archive, or sync behavior.
- Apply does not stop/start Gitea, create a backup container, call Google Drive,
prune snapshots, restart a VM or service, change a database/network/firewall,
or read/change a credential or secret.
## Reproducible Validation
Expanded focused and compatibility suite:
```sh
pytest -q \
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
scripts/backup/tests/test_verify_offsite_full_sync_runtime.py \
scripts/backup/tests/test_verify_host188_products_archive.py \
scripts/backup/tests/test_backup_host188_products_contract.py \
scripts/ops/tests/test_agent99_signoz_metadata_executor_runtime_entrypoint.py \
scripts/ops/tests/test_db_bounded_executor_runtime_entrypoint.py \
scripts/reboot-recovery/tests/test_agent99_live_preflight_decision.py \
scripts/reboot-recovery/tests/test_agent99_remote_atomic_deploy_transport_contract.py \
scripts/reboot-recovery/tests/test_host112_manager_recovery_contract.py
```
Current result: `120 passed, 4 skipped`.
The shared-lock subset includes real `fcntl` contention for absolute,
relative, symlink, forged-environment, inherited-exact-FD, inherited-wrong-FD,
and Host188 archive paths. Current result: `18 passed, 31 deselected`.
Static, syntax, and repository checks:
```sh
ruff check \
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
scripts/backup/verify-host188-products-archive.py
bash -n \
scripts/backup/common.sh \
scripts/backup/gitea-full-backup-restore-drill.sh \
scripts/backup/backup-offsite-readiness-gate.sh \
scripts/backup/host110-backup-runtime-executor.sh
git diff --check
PATH=/Users/ogt/.pyenv/shims:$PATH bash scripts/ops/ansible-validate.sh
```
`ansible-validate.sh` parses
`ops/reboot-recovery/full-stack-backup-baseline.yml` as YAML metadata. It runs
`ansible-playbook --syntax-check` only for actual playbooks. Current checks are
green; `ansible-lint` is not installed and is reported as skipped.
Windows99 no-write PowerShell validation consists of:
1. parsing the complete candidate broker with Windows PowerShell;
2. loading the candidate broker's actual function AST in the contract replay;
3. replaying valid success, committed non-success, existing evidence rejection,
terminal-missing reconciliation, and committed-unknown cases.
Current readback is parser `errorCount=0`, plus 38 existing-evidence
rejections, 38 missing-field rejections, 6 type rejections,
`terminalMissingReconciled=true`, `committedUnknownPreserved=true`,
`remoteWritePerformed=false`, and `secretValuesRead=false`.
## Exact Identity And Diff Serialization
The central-review intake must bind all of the following independently:
- live Gitea feature ref `codex/host110-backup-runtime-v11-20260718`;
- exact candidate Git SHA;
- this tracked artifact path and exact artifact-bytes SHA-256;
- exact binary diff SHA-256, path count, and insertion/deletion counts;
- the 19/18/21 count contracts;
- `scripts/backup/backup-gitea.sh` zero diff;
- the Windows99 no-write replay result.
Exact diff bytes are serialized with:
```sh
git diff --binary --full-index \
e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3..<candidate_git_sha> |
shasum -a 256
```
No alternative diff serialization or untracked artifact bytes may be used for
approval identity.
## Rollback
- Source rollback reverts the exact V11 diff to
`e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3` without force-pushing.
- Agent99 rollback restores prior runtime files and manifest and verifies prior
hashes independently.
- Before payload commit, Host110 rollback restores prior payload content,
SHA-256, uid, gid, and mode under the exclusive lock and proves no run-owned
destination temporary remains.
- After payload commit, transport loss cannot trigger a blind Apply retry.
Exact receipts and independent destination readback determine whether the
state is committed or unknown; every uncertain state remains non-success.
- Source rollback restores prior lock behavior. It does not execute a backup,
restore drill, offsite sync, container lifecycle, or service restart.
- Any identity, schema, type, receipt, reconciliation, lock, cleanup, or
residue mismatch fails closed and cannot produce broker `verified`.

View File

@@ -0,0 +1,78 @@
# AWOOOI Agent99 Host110 Backup Runtime V8
Status: proposed; owner approval required before any production apply.
This artifact supersedes V2 through V7. The V7 candidate, revision, artifact,
and diff hashes are void for approval. All V5 hashes remain void. Prior refs
remain immutable audit evidence only; none is production execution authority.
## Scope
- Agent99 runtime bundle: 19 files, guarded by one producer/consumer parity
test.
- Host110 backup payload: 18 files, comprising 17 backup scripts and
`backup-health-textfile-exporter.py`.
- Host110 remote stage: 21 unique basenames, comprising the 18 payload files,
the fixed executor, the independent verifier, and `manifest.json`.
- Fixed target: `wooo@192.168.0.110`, dispatched by Windows99 Agent99 from an
exact Gitea main revision and digest manifest.
- Gitea offline backup and container lifecycle changes remain excluded.
`backup-gitea.sh` has no candidate diff.
## V7 Defect Closed
V7 created and uploaded
`agent99-host110-backup-runtime-manifest.json`, while the fixed Host110
executor consumed `$SOURCE_STAGE/manifest.json`. V8 uses `manifest.json` at
the broker source stage, so SCP preserves the exact basename consumed by the
executor.
The integration test derives all broker SCP basenames from the actual
PowerShell file list and manifest path, requires 21 unique stage files, and
matches the manifest and verifier basenames to the executor's exact
`$SOURCE_STAGE` paths. A hand-built temporary manifest cannot satisfy this
test by itself.
## Exact Diff Serialization
Substitute the V8 Gitea live-ref SHA for `<candidate_git_sha>` and hash the
exact stdout bytes from this command:
```sh
git diff --binary --full-index \
e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3..<candidate_git_sha> |
shasum -a 256
```
The review message independently binds the Gitea live ref and Git SHA, this
tracked artifact path and bytes hash, and the serialized diff hash.
## Apply Effects
- Windows99 Agent99 runtime promotion writes the 19 runtime files plus bounded
manifest and receipt state.
- Host110 Apply writes/replaces the 18 payload paths through one exclusive,
digest-bound filesystem transaction.
- Apply does not run a backup, invoke a Gitea container lifecycle, sync Google
Drive, prune or delete snapshots, restart a VM or service, or change a DB,
network, or firewall.
## Validation
- The exact broker/SCP basename to executor path integration test must pass.
- Focused runtime, broker, backup, parity, and Ansible suites must pass.
- Ansible and shell/Python static validation must pass.
- Changed PowerShell must parse on Windows99 with zero errors.
- Central review must independently reproduce the live ref, artifact hash,
diff hash, 19/18/21 counts, and zero `backup-gitea.sh` diff.
## Rollback
- Source rollback reverts the exact V8 diff to
`e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3` without force-pushing.
- Agent99 rollback restores the prior runtime files and manifest and verifies
the prior hashes independently.
- Host110 rollback restores every prior payload's content, SHA-256, uid, gid,
and mode under the exclusive lock.
- Any mismatch terminates as `rollback_unverified` and cannot be reported as a
successful rollback.

View File

@@ -0,0 +1,133 @@
# AWOOOI Agent99 Host110 Backup Runtime V9
Status: proposed; owner approval required before any production apply.
This artifact supersedes V2 through V8. The following V8 identities are void
for approval and remain audit evidence only:
- candidate/revision: `0b9c0284006c8d892c4617c2bfc16869bb6637f0`
- artifact SHA-256: `ea9537613c462a07f3155263da5ad0f444bda0ba6773b49208de31b0acdef228`
- exact diff SHA-256: `0867d9221ecfa7bcbe6b073a6a02da401bbed5087b9f2bd7fa494f2953b2ef2e`
- every other proposal/content hash derived from V8 candidate bytes
No V8 hash, ref, review, or receipt grants production execution authority.
## Scope
- Agent99 runtime bundle: 19 files, guarded by one producer/consumer parity
test.
- Host110 backup payload: 18 ordered files, comprising 17 backup scripts and
`backup-health-textfile-exporter.py`.
- Host110 remote stage: 21 unique basenames, comprising the ordered 18-file
payload, fixed executor, independent verifier, and `manifest.json`.
- Fixed target: `wooo@192.168.0.110`, dispatched by Windows99 Agent99 from an
exact Gitea revision and digest manifest.
- Gitea offline backup and container lifecycle changes remain excluded;
`backup-gitea.sh` has no candidate diff from the production base.
## V8 Defects Closed
### Durable Receipt Transaction
The executor no longer captures `write_receipt` through command substitution
and cannot mask an internal failure with a later `printf`. Receipt publication
uses an exclusive temporary file, exact full-write check, file `fsync`,
temporary-byte SHA-256 readback, no-replace hard link, directory `fsync`, exact
final-byte/document/SHA-256 readback, temporary unlink, and a second directory
`fsync`. Any write, link, fsync, cleanup, or readback failure is nonzero and
removes a final path created by that failed attempt when possible.
`APPLY_COMPLETE=1` is set only after the durable verified receipt passes this
transaction. Stage and rollback prestate are never cleared before that point.
If verified receipt publication fails, the EXIT path performs rollback first,
writes a distinct failure receipt, and only removes rollback prestate after
both rollback verification and durable failure-receipt publication succeed.
Payload files, destination directories, rollback prestate files, and the
rollback directory are explicitly fsynced before their corresponding durable
claim can be published.
### Run-Owned Temporary Files
Every apply and rollback temporary destination is tracked for the exact run.
Rollback removes and reads back all tracked paths, including a failure between
`install` and `mv`. Canonical content/metadata verification plus zero hidden
residue are both required before rollback can be called verified.
### Rollback Truth
Receipt schema `agent99_host110_backup_runtime_deploy_receipt_v2` records
`rollbackAttempted`, `rollbackPerformed`, `rollbackVerified`, and
`zeroResidueVerified` independently. A partial, unknown, or residue-bearing
rollback terminates as `rollback_unverified`; only a fully restored and
zero-residue transaction may terminate as `failed_rolled_back`.
### Producer/Consumer Parity
The integration test parses both actual sources and requires the broker's 18
payload basenames to equal the executor payload in count, exact set, and exact
order. It separately requires executor, verifier, and `manifest.json`, yielding
21 unique remote-stage basenames.
## Fault-Injection Contract
The bounded local harness executes the real executor control flow against
isolated temporary destinations and covers:
- successful apply, durable receipt, verifier, and post-receipt prestate cleanup;
- receipt write, link, fsync, and final-readback failures;
- apply failure after hidden-temp installation but before canonical `mv`;
- forced temp-cleanup failure and `rollback_unverified` evidence preservation.
Every receipt fault must restore exact prestate, emit no success JSON, and
produce a durable `failed_rolled_back` receipt. A cleanup fault must preserve
`prestate.tsv`, expose rollback attempted/performed separately, and must not
claim rollback verification.
Fault hooks require an explicit test enable flag and a transformed
non-production destination. The canonical `/backup/scripts` executor can never
enable them, even if a fault environment variable is present.
## Exact Diff Serialization
Substitute the V9 Gitea live-ref SHA for `<candidate_git_sha>` and hash the
exact stdout bytes from this command:
```sh
git diff --binary --full-index \
e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3..<candidate_git_sha> |
shasum -a 256
```
The review message independently binds the Gitea live ref and Git SHA, this
tracked artifact path and bytes hash, and the serialized diff hash.
## Apply Effects
- Windows99 Agent99 runtime promotion writes the 19 runtime files plus bounded
manifest and receipt state.
- Host110 Apply writes/replaces the 18 payload paths through one exclusive,
digest-bound filesystem transaction.
- Apply does not run a backup, invoke a Gitea container lifecycle, sync Google
Drive, prune snapshots, restart a VM or service, or change a database,
network, or firewall.
## Validation
- The exact broker payload/order and 21-file stage parity test must pass.
- Success and all six fault-injection paths must execute and pass.
- Focused runtime, broker, backup, parity, Ansible, shell, and Python checks
must pass.
- Changed PowerShell must parse on Windows99 with zero errors when applicable.
- Central review must independently reproduce the live ref, artifact hash,
exact diff hash, 19/18/21 counts, and zero `backup-gitea.sh` diff.
## Rollback
- Source rollback reverts the exact V9 diff to
`e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3` without force-pushing.
- Agent99 rollback restores the prior runtime files and manifest and verifies
prior hashes independently.
- Host110 rollback restores all prior payload content, SHA-256, uid, gid, and
mode under the exclusive lock and proves zero run-owned temporary residue.
- Any canonical, metadata, receipt, or residue mismatch terminates as
`rollback_unverified`; rollback prestate remains available for diagnosis.

View File

@@ -9,14 +9,74 @@
set -euo pipefail
if [[ "${BASH_SOURCE[0]}" == /backup/scripts/* ]] && [ "${BACKUP_RUNTIME_SHARED_LOCK_HELD:-0}" != "1" ]; then
resolve_backup_runtime_script_path() {
local raw_path="$1" candidate="" resolved="" resolved_directory=""
[ -n "${raw_path}" ] || return 1
case "${raw_path}" in /*) candidate="${raw_path}" ;; *) candidate="${PWD}/${raw_path}" ;; esac
if command -v realpath >/dev/null 2>&1; then
resolved="$(realpath -e -- "${candidate}" 2>/dev/null || realpath -- "${candidate}" 2>/dev/null)" || resolved=""
if [ -n "${resolved}" ]; then printf '%s\n' "${resolved}"; return 0; fi
fi
if command -v readlink >/dev/null 2>&1; then
resolved="$(readlink -f -- "${candidate}" 2>/dev/null)" || resolved=""
if [ -n "${resolved}" ]; then printf '%s\n' "${resolved}"; return 0; fi
fi
[ ! -L "${candidate}" ] || return 1
resolved_directory="$(cd -P -- "$(dirname -- "${candidate}")" 2>/dev/null && pwd -P)" || return 1
[ -e "${resolved_directory}/$(basename -- "${candidate}")" ] || return 1
printf '%s/%s\n' "${resolved_directory}" "$(basename -- "${candidate}")"
}
backup_runtime_inherited_fd_status() {
local expected_path="$1" fd_path="" fd_target=""
if [ -d "/proc/$$/fd" ]; then
fd_path="/proc/$$/fd/197"
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
fd_target="$(resolve_backup_runtime_script_path "${fd_path}")" || return 2
[ "${fd_target}" = "${expected_path}" ] || return 2
return 0
fi
fd_path="/dev/fd/197"
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
command -v python3 >/dev/null 2>&1 || return 2
fd_target="$(python3 - <<'PY'
import fcntl
import os
try:
target = os.readlink("/dev/fd/197")
except OSError:
value = fcntl.fcntl(197, getattr(fcntl, "F_GETPATH", 50), b"\0" * 1024)
target = os.fsdecode(value.split(b"\0", 1)[0])
print(os.path.realpath(target))
PY
)" || return 2
[ "${fd_target}" = "${expected_path}" ] || return 2
}
runtime_script_path="$(resolve_backup_runtime_script_path "${BASH_SOURCE[0]:-}")" || {
echo "backup runtime gate unavailable: script identity unresolved" >&2
exit 69
}
if [[ "${runtime_script_path}" == /backup/scripts/* ]]; then
command -v flock >/dev/null 2>&1 || { echo "backup runtime gate unavailable" >&2; exit 69; }
[ ! -L /tmp/agent99-host110-backup-runtime.lock ] || { echo "backup runtime gate unsafe" >&2; exit 69; }
(umask 077; : >> /tmp/agent99-host110-backup-runtime.lock)
[ -f /tmp/agent99-host110-backup-runtime.lock ] && [ -O /tmp/agent99-host110-backup-runtime.lock ] || { echo "backup runtime gate owner invalid" >&2; exit 69; }
exec 197>>/tmp/agent99-host110-backup-runtime.lock
runtime_lock_path="$(resolve_backup_runtime_script_path /tmp/agent99-host110-backup-runtime.lock)" \
|| { echo "backup runtime gate unavailable: lock identity unresolved" >&2; exit 69; }
if backup_runtime_inherited_fd_status "${runtime_lock_path}"; then
:
else
inherited_fd_status=$?
if [ "${inherited_fd_status}" -eq 1 ]; then
exec 197>>/tmp/agent99-host110-backup-runtime.lock
else
echo "backup runtime gate unsafe: inherited fd 197 identity mismatch" >&2
exit 69
fi
fi
flock -s -w 60 197 || { echo "backup runtime deployment is active; retry later" >&2; exit 75; }
export BACKUP_RUNTIME_SHARED_LOCK_HELD=1
fi
BACKUP_BASE="${BACKUP_BASE:-/backup}"

View File

@@ -3,14 +3,78 @@
# Production backup entrypoints share this stable gate. Agent99 takes the
# exclusive side while promoting a complete, digest-bound runtime bundle.
resolve_backup_runtime_source_path() {
local raw_path="$1"
local candidate=""
local resolved=""
local resolved_directory=""
[ -n "${raw_path}" ] || return 1
case "${raw_path}" in
/*) candidate="${raw_path}" ;;
*) candidate="${PWD}/${raw_path}" ;;
esac
if command -v realpath >/dev/null 2>&1; then
resolved="$(realpath -e -- "${candidate}" 2>/dev/null || realpath -- "${candidate}" 2>/dev/null)" || resolved=""
if [ -n "${resolved}" ]; then
printf '%s\n' "${resolved}"
return 0
fi
fi
if command -v readlink >/dev/null 2>&1; then
resolved="$(readlink -f -- "${candidate}" 2>/dev/null)" || resolved=""
if [ -n "${resolved}" ]; then
printf '%s\n' "${resolved}"
return 0
fi
fi
[ ! -L "${candidate}" ] || return 1
resolved_directory="$(cd -P -- "$(dirname -- "${candidate}")" 2>/dev/null && pwd -P)" || return 1
[ -e "${resolved_directory}/$(basename -- "${candidate}")" ] || return 1
printf '%s/%s\n' "${resolved_directory}" "$(basename -- "${candidate}")"
}
backup_runtime_inherited_fd_status() {
local expected_path="$1"
local fd_path=""
local fd_target=""
if [ -d "/proc/$$/fd" ]; then
fd_path="/proc/$$/fd/197"
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
fd_target="$(resolve_backup_runtime_source_path "${fd_path}")" || return 2
[ "${fd_target}" = "${expected_path}" ] || return 2
return 0
fi
fd_path="/dev/fd/197"
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
command -v python3 >/dev/null 2>&1 || return 2
fd_target="$(python3 - <<'PY'
import fcntl
import os
try:
target = os.readlink("/dev/fd/197")
except OSError:
value = fcntl.fcntl(197, getattr(fcntl, "F_GETPATH", 50), b"\0" * 1024)
target = os.fsdecode(value.split(b"\0", 1)[0])
print(os.path.realpath(target))
PY
)" || return 2
[ "${fd_target}" = "${expected_path}" ] || return 2
}
acquire_backup_runtime_shared_lock() {
local caller_path="${BASH_SOURCE[1]:-${BASH_SOURCE[0]}}"
local caller_path=""
local lock_path="/tmp/agent99-host110-backup-runtime.lock"
local lock_canonical=""
local inherited_fd_status=0
caller_path="$(resolve_backup_runtime_source_path "${BASH_SOURCE[1]:-${BASH_SOURCE[0]:-}}")" || {
echo "backup runtime gate unavailable: caller identity unresolved" >&2
return 69
}
case "${caller_path}" in
/backup/scripts/*) ;;
*) return 0 ;;
esac
[ "${BACKUP_RUNTIME_SHARED_LOCK_HELD:-0}" != "1" ] || return 0
command -v flock >/dev/null 2>&1 || {
echo "backup runtime gate unavailable: flock missing" >&2
return 69
@@ -21,12 +85,25 @@ acquire_backup_runtime_shared_lock() {
}
(umask 077; : >> "${lock_path}") || return 69
[ -f "${lock_path}" ] && [ -O "${lock_path}" ] || return 69
exec 197>>"${lock_path}"
lock_canonical="$(resolve_backup_runtime_source_path "${lock_path}")" || {
echo "backup runtime gate unavailable: lock identity unresolved" >&2
return 69
}
if backup_runtime_inherited_fd_status "${lock_canonical}"; then
:
else
inherited_fd_status=$?
if [ "${inherited_fd_status}" -eq 1 ]; then
exec 197>>"${lock_path}"
else
echo "backup runtime gate unsafe: inherited fd 197 identity mismatch" >&2
return 69
fi
fi
flock -s -w 60 197 || {
echo "backup runtime deployment is active; retry later" >&2
return 75
}
export BACKUP_RUNTIME_SHARED_LOCK_HELD=1
}
acquire_backup_runtime_shared_lock || {

View File

@@ -5,14 +5,74 @@
set -euo pipefail
if [[ "${BASH_SOURCE[0]}" == /backup/scripts/* ]] && [ "${BACKUP_RUNTIME_SHARED_LOCK_HELD:-0}" != "1" ]; then
resolve_backup_runtime_script_path() {
local raw_path="$1" candidate="" resolved="" resolved_directory=""
[ -n "${raw_path}" ] || return 1
case "${raw_path}" in /*) candidate="${raw_path}" ;; *) candidate="${PWD}/${raw_path}" ;; esac
if command -v realpath >/dev/null 2>&1; then
resolved="$(realpath -e -- "${candidate}" 2>/dev/null || realpath -- "${candidate}" 2>/dev/null)" || resolved=""
if [ -n "${resolved}" ]; then printf '%s\n' "${resolved}"; return 0; fi
fi
if command -v readlink >/dev/null 2>&1; then
resolved="$(readlink -f -- "${candidate}" 2>/dev/null)" || resolved=""
if [ -n "${resolved}" ]; then printf '%s\n' "${resolved}"; return 0; fi
fi
[ ! -L "${candidate}" ] || return 1
resolved_directory="$(cd -P -- "$(dirname -- "${candidate}")" 2>/dev/null && pwd -P)" || return 1
[ -e "${resolved_directory}/$(basename -- "${candidate}")" ] || return 1
printf '%s/%s\n' "${resolved_directory}" "$(basename -- "${candidate}")"
}
backup_runtime_inherited_fd_status() {
local expected_path="$1" fd_path="" fd_target=""
if [ -d "/proc/$$/fd" ]; then
fd_path="/proc/$$/fd/197"
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
fd_target="$(resolve_backup_runtime_script_path "${fd_path}")" || return 2
[ "${fd_target}" = "${expected_path}" ] || return 2
return 0
fi
fd_path="/dev/fd/197"
[ -e "${fd_path}" ] || [ -L "${fd_path}" ] || return 1
command -v python3 >/dev/null 2>&1 || return 2
fd_target="$(python3 - <<'PY'
import fcntl
import os
try:
target = os.readlink("/dev/fd/197")
except OSError:
value = fcntl.fcntl(197, getattr(fcntl, "F_GETPATH", 50), b"\0" * 1024)
target = os.fsdecode(value.split(b"\0", 1)[0])
print(os.path.realpath(target))
PY
)" || return 2
[ "${fd_target}" = "${expected_path}" ] || return 2
}
runtime_script_path="$(resolve_backup_runtime_script_path "${BASH_SOURCE[0]:-}")" || {
echo "backup runtime gate unavailable: script identity unresolved" >&2
exit 69
}
if [[ "${runtime_script_path}" == /backup/scripts/* ]]; then
command -v flock >/dev/null 2>&1 || { echo "backup runtime gate unavailable" >&2; exit 69; }
[ ! -L /tmp/agent99-host110-backup-runtime.lock ] || { echo "backup runtime gate unsafe" >&2; exit 69; }
(umask 077; : >> /tmp/agent99-host110-backup-runtime.lock)
[ -f /tmp/agent99-host110-backup-runtime.lock ] && [ -O /tmp/agent99-host110-backup-runtime.lock ] || { echo "backup runtime gate owner invalid" >&2; exit 69; }
exec 197>>/tmp/agent99-host110-backup-runtime.lock
runtime_lock_path="$(resolve_backup_runtime_script_path /tmp/agent99-host110-backup-runtime.lock)" \
|| { echo "backup runtime gate unavailable: lock identity unresolved" >&2; exit 69; }
if backup_runtime_inherited_fd_status "${runtime_lock_path}"; then
:
else
inherited_fd_status=$?
if [ "${inherited_fd_status}" -eq 1 ]; then
exec 197>>/tmp/agent99-host110-backup-runtime.lock
else
echo "backup runtime gate unsafe: inherited fd 197 identity mismatch" >&2
exit 69
fi
fi
flock -s -w 60 197 || { echo "backup runtime deployment is active; retry later" >&2; exit 75; }
export BACKUP_RUNTIME_SHARED_LOCK_HELD=1
fi
DUMP_ZIP="${AIOPS_GITEA_FULL_BACKUP_DRILL_DUMP_ZIP:-}"

View File

@@ -44,13 +44,23 @@ EXECUTOR_DIGEST=""
VERIFIER_DIGEST=""
STAGE_DIR=""
ROLLBACK_DIR=""
RECEIPT_PATH=""
TERMINAL_RECEIPT_PATH=""
APPLY_STARTED=0
APPLY_COMPLETE=0
COMMIT_CRITICAL=0
DEFERRED_SIGNAL=""
DEFERRED_SIGNAL_EXIT=0
ROLLBACK_ATTEMPTED=0
ROLLBACK_PERFORMED=0
ROLLBACK_VERIFIED=0
RUN_TEMP_RESIDUE_VERIFIED=0
FAULT_INJECTION_ENABLED=0
declare -A FILE_EXISTED=()
declare -A PREVIOUS_DIGEST=()
declare -A PREVIOUS_METADATA=()
declare -A EXPECTED_DIGEST=()
declare -A RUN_OWNED_TEMP_PATHS=()
usage() {
printf '%s\n' \
@@ -96,6 +106,18 @@ case "$MODE" in check|apply|verify) ;; *) fail "invalid_mode" ;; esac
[[ "$RUN_ID" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,95}$ ]] || fail "invalid_run_id"
expected_source_stage="/tmp/agent99-host110-backup-runtime-${RUN_ID}"
[ "$SOURCE_STAGE" = "$expected_source_stage" ] || fail "invalid_source_stage"
RECEIPT_PATH="$STATUS_ROOT/agent99-backup-runtime-deploy-${RUN_ID}.json"
TERMINAL_RECEIPT_PATH="$STATUS_ROOT/agent99-backup-runtime-terminal-${RUN_ID}.json"
if [ "${HOST110_BACKUP_RUNTIME_ENABLE_TEST_FAULTS:-}" = "1" ] \
&& [ "$DEST_ROOT" != "/backup/scripts" ]; then
FAULT_INJECTION_ENABLED=1
fi
if [ "$FAULT_INJECTION_ENABLED" -eq 1 ]; then
case "${HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL:-}" in ""|INT|HUP|TERM) ;; *) fail "invalid_test_commit_signal" ;; esac
case "${HOST110_BACKUP_RUNTIME_TEST_TERMINAL_SIGNAL:-}" in ""|INT|HUP|TERM) ;; *) fail "invalid_test_terminal_signal" ;; esac
case "${HOST110_BACKUP_RUNTIME_TEST_TERMINAL_RECEIPT_FAULT:-}" in ""|write|link|fsync|readback) ;; *) fail "invalid_test_terminal_receipt_fault" ;; esac
case "${HOST110_BACKUP_RUNTIME_TEST_CLEANUP_FAULT:-}" in ""|stage|rollback_prestate) ;; *) fail "invalid_test_cleanup_fault" ;; esac
fi
for command_name in awk bash cp dirname flock grep hostname id install mv pgrep python3 readlink rm sha256sum stat timeout tr; do
command -v "$command_name" >/dev/null 2>&1 || fail "required_command_missing_${command_name}"
@@ -174,10 +196,10 @@ working_digest() {
|| fail "verifier_identity_failed"
validate_file() {
local path="$1"
case "$path" in
*.sh) bash -n "$path" ;;
*.py) python3 -c 'import ast, pathlib, sys; ast.parse(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))' "$path" ;;
local validation_path="$1"
case "$validation_path" in
*.sh) bash -n "$validation_path" ;;
*.py) python3 -c 'import ast, pathlib, sys; ast.parse(pathlib.Path(sys.argv[1]).read_text(encoding="utf-8"))' "$validation_path" ;;
*) return 64 ;;
esac
}
@@ -222,12 +244,75 @@ verify_destination() {
done
}
fsync_paths_and_parents() {
python3 - "$@" <<'PY'
import os
import stat
import sys
from pathlib import Path
parents = set()
for raw_path in sys.argv[1:]:
path = Path(raw_path)
parents.add(path.parent)
if not os.path.lexists(path):
continue
if path.is_symlink() or not path.is_file():
raise SystemExit(76)
flags = os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0)
descriptor = os.open(path, flags)
try:
if not stat.S_ISREG(os.fstat(descriptor).st_mode):
raise SystemExit(76)
os.fsync(descriptor)
finally:
os.close(descriptor)
for parent in sorted(parents, key=str):
if parent.is_symlink() or not parent.is_dir():
raise SystemExit(76)
flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0)
descriptor = os.open(parent, flags)
try:
os.fsync(descriptor)
finally:
os.close(descriptor)
PY
}
fsync_destination_state() {
local name dest_path
local -a fsync_targets=()
for name in "${PAYLOAD_FILES[@]}"; do
dest_path="$(destination_path "$name")"
fsync_targets+=("$dest_path")
done
fsync_paths_and_parents "${fsync_targets[@]}"
}
fsync_rollback_prestate() {
local name
local -a fsync_targets=("$ROLLBACK_DIR/prestate.tsv")
for name in "${PAYLOAD_FILES[@]}"; do
if [ "${FILE_EXISTED[$name]:-}" = "1" ]; then
fsync_targets+=("$ROLLBACK_DIR/$name")
fi
done
fsync_paths_and_parents "${fsync_targets[@]}"
}
write_receipt() {
local status="$1"
local rollback_verified="$2"
local receipt_path="$STATUS_ROOT/agent99-backup-runtime-deploy-${RUN_ID}.json"
install -d -m 700 "$STATUS_ROOT"
python3 - "$receipt_path" "$status" "$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID" "$rollback_verified" "$VERIFIER_DIGEST" <<'PY'
local receipt_status="$1"
local rollback_attempted="$2"
local rollback_performed="$3"
local rollback_verified="$4"
local zero_residue_verified="$5"
install -d -m 700 "$STATUS_ROOT" || return 1
[ -d "$STATUS_ROOT" ] && [ ! -L "$STATUS_ROOT" ] || return 1
[ "$(stat -c '%U:%G:%a' "$STATUS_ROOT")" = "wooo:wooo:700" ] || return 1
if ! python3 - "$RECEIPT_PATH" "$receipt_status" "$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID" \
"$rollback_attempted" "$rollback_performed" "$rollback_verified" \
"$zero_residue_verified" "$VERIFIER_DIGEST" "$FAULT_INJECTION_ENABLED" <<'PY'
import hashlib
import json
import os
import sys
@@ -235,64 +320,343 @@ import time
from pathlib import Path
path = Path(sys.argv[1])
status = sys.argv[2]
rollback_attempted = sys.argv[6] == "1"
rollback_performed = sys.argv[7] == "1"
rollback_verified = sys.argv[8] == "1"
zero_residue_verified = sys.argv[9] == "1"
if status not in {"payload_verified", "failed_rolled_back", "rollback_unverified"}:
raise SystemExit(78)
if status == "payload_verified" and (rollback_attempted or rollback_performed or rollback_verified or not zero_residue_verified):
raise SystemExit(78)
if status == "failed_rolled_back" and not (
rollback_attempted and rollback_performed and rollback_verified and zero_residue_verified
):
raise SystemExit(78)
if status == "rollback_unverified" and (not rollback_attempted or rollback_verified):
raise SystemExit(78)
document = {
"schemaVersion": "agent99_host110_backup_runtime_deploy_receipt_v1",
"status": sys.argv[2],
"schemaVersion": "agent99_host110_backup_runtime_deploy_receipt_v3",
"status": status,
"sourceRevision": sys.argv[3],
"sourceHead": sys.argv[4],
"runId": sys.argv[5],
"fileCount": 18,
"backupScriptFileCount": 17,
"backupHealthExporterIncluded": True,
"rollbackPerformed": sys.argv[6] == "1",
"rollbackVerified": sys.argv[6] == "1",
"verifierSha256": sys.argv[7],
"rollbackAttempted": rollback_attempted,
"rollbackPerformed": rollback_performed,
"rollbackVerified": rollback_verified,
"zeroResidueVerified": zero_residue_verified,
"zeroResidueScope": "run_owned_destination_temps",
"verifierSha256": sys.argv[10],
"executorHost": "192.168.0.110",
"productionServiceRestarted": False,
"secretValuesRead": False,
"writtenAt": int(time.time()),
}
payload = (json.dumps(document, ensure_ascii=True, sort_keys=True, separators=(",", ":")) + "\n").encode("utf-8")
expected_digest = hashlib.sha256(payload).hexdigest()
temporary = path.with_name(f".{path.name}.tmp-{os.getpid()}")
temporary.write_text(json.dumps(document, ensure_ascii=True, sort_keys=True) + "\n", encoding="utf-8")
os.chmod(temporary, 0o600)
fault_enabled = sys.argv[11] == "1"
fault = (
os.environ.get("HOST110_BACKUP_RUNTIME_TEST_RECEIPT_FAULT", "")
if status == "payload_verified" and fault_enabled
else ""
)
if fault not in {"", "write", "link", "fsync", "readback"}:
raise SystemExit(78)
created_final = False
directory_fd = None
try:
if os.path.lexists(path) or os.path.lexists(temporary):
raise FileExistsError(path)
if fault == "write":
raise OSError("fault_injected_receipt_write")
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(descriptor, "wb") as handle:
if handle.write(payload) != len(payload):
raise OSError("receipt_short_write")
handle.flush()
if fault == "fsync":
raise OSError("fault_injected_receipt_fsync")
os.fsync(handle.fileno())
os.chmod(temporary, 0o600)
temporary_readback = temporary.read_bytes()
if temporary_readback != payload or hashlib.sha256(temporary_readback).hexdigest() != expected_digest:
raise OSError("receipt_temporary_readback_failed")
if fault == "link":
raise OSError("fault_injected_receipt_link")
os.link(temporary, path)
created_final = True
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
directory_fd = os.open(path.parent, directory_flags)
os.fsync(directory_fd)
if fault == "readback":
raise OSError("fault_injected_receipt_readback")
readback = path.read_bytes()
if readback != payload or hashlib.sha256(readback).hexdigest() != expected_digest:
raise OSError("receipt_final_readback_failed")
if json.loads(readback.decode("utf-8")) != document:
raise OSError("receipt_document_readback_failed")
temporary.unlink()
os.fsync(directory_fd)
except BaseException:
try:
if created_final:
path.unlink(missing_ok=True)
temporary.unlink(missing_ok=True)
if directory_fd is None and path.parent.is_dir():
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
directory_fd = os.open(path.parent, directory_flags)
if directory_fd is not None:
os.fsync(directory_fd)
finally:
raise
finally:
temporary.unlink(missing_ok=True)
if directory_fd is not None:
os.close(directory_fd)
PY
printf '%s' "$receipt_path"
then
return 1
fi
[ -f "$RECEIPT_PATH" ] && [ ! -L "$RECEIPT_PATH" ] || return 1
return 0
}
write_terminal_receipt() {
local terminal_status="$1"
local stage_cleanup_verified="$2"
local rollback_prestate_cleanup_verified="$3"
local deferred_signal="$4"
local terminal_exit_code="$5"
install -d -m 700 "$STATUS_ROOT" || return 1
[ -d "$STATUS_ROOT" ] && [ ! -L "$STATUS_ROOT" ] || return 1
[ "$(stat -c '%U:%G:%a' "$STATUS_ROOT")" = "wooo:wooo:700" ] || return 1
if ! python3 - "$TERMINAL_RECEIPT_PATH" "$RECEIPT_PATH" "$terminal_status" \
"$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID" "$stage_cleanup_verified" \
"$rollback_prestate_cleanup_verified" "$deferred_signal" "$terminal_exit_code" \
"$VERIFIER_DIGEST" "$FAULT_INJECTION_ENABLED" <<'PY'
import hashlib
import json
import os
import signal
import sys
import time
from pathlib import Path
path = Path(sys.argv[1])
payload_receipt_path = Path(sys.argv[2])
status = sys.argv[3]
source_revision = sys.argv[4]
source_head = sys.argv[5]
run_id = sys.argv[6]
stage_cleanup_verified = sys.argv[7] == "1"
rollback_prestate_cleanup_verified = sys.argv[8] == "1"
deferred_signal = sys.argv[9]
terminal_exit_code = int(sys.argv[10])
verifier_digest = sys.argv[11]
fault_enabled = sys.argv[12] == "1"
signal_exit_codes = {"INT": 130, "HUP": 129, "TERM": 143}
terminal_signal = os.environ.get("HOST110_BACKUP_RUNTIME_TEST_TERMINAL_SIGNAL", "") if fault_enabled else ""
terminal_fault = os.environ.get("HOST110_BACKUP_RUNTIME_TEST_TERMINAL_RECEIPT_FAULT", "") if fault_enabled else ""
if terminal_signal:
os.kill(os.getppid(), getattr(signal, f"SIG{terminal_signal}"))
time.sleep(0.05)
try:
payload_receipt = json.loads(payload_receipt_path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
raise SystemExit(79)
if not (
payload_receipt.get("schemaVersion") == "agent99_host110_backup_runtime_deploy_receipt_v3"
and payload_receipt.get("status") == "payload_verified"
and payload_receipt.get("sourceRevision") == source_revision
and payload_receipt.get("sourceHead") == source_head
and payload_receipt.get("runId") == run_id
and payload_receipt.get("zeroResidueVerified") is True
):
raise SystemExit(79)
if status == "verified":
valid = (
stage_cleanup_verified
and rollback_prestate_cleanup_verified
and not deferred_signal
and terminal_exit_code == 0
)
elif status == "cleanup_pending":
valid = (
not (stage_cleanup_verified and rollback_prestate_cleanup_verified)
and deferred_signal in {"", *signal_exit_codes}
and terminal_exit_code != 0
)
elif status == "committed_signal_deferred":
valid = (
stage_cleanup_verified
and rollback_prestate_cleanup_verified
and deferred_signal in signal_exit_codes
and terminal_exit_code == signal_exit_codes[deferred_signal]
)
else:
valid = False
if not valid:
raise SystemExit(79)
document = {
"schemaVersion": "agent99_host110_backup_runtime_terminal_receipt_v1",
"status": status,
"ok": status == "verified",
"payloadCommitted": True,
"payloadReceipt": str(payload_receipt_path),
"sourceRevision": source_revision,
"sourceHead": source_head,
"runId": run_id,
"fileCount": 18,
"backupScriptFileCount": 17,
"backupHealthExporterIncluded": True,
"stageCleanupVerified": stage_cleanup_verified,
"rollbackPrestateCleanupVerified": rollback_prestate_cleanup_verified,
"cleanupVerified": stage_cleanup_verified and rollback_prestate_cleanup_verified,
"independentVerifierVerified": True,
"zeroResidueScope": "run_owned_destination_temps_and_internal_stage_prestate",
"deferredSignal": deferred_signal or None,
"terminalExitCode": terminal_exit_code,
"verifierSha256": verifier_digest,
"executorHost": "192.168.0.110",
"productionServiceRestarted": False,
"secretValuesRead": False,
"writtenAt": int(time.time()),
}
payload = (json.dumps(document, ensure_ascii=True, sort_keys=True, separators=(",", ":")) + "\n").encode("utf-8")
expected_digest = hashlib.sha256(payload).hexdigest()
temporary = path.with_name(f".{path.name}.tmp-{os.getpid()}")
created_final = False
directory_fd = None
try:
if os.path.lexists(path) or os.path.lexists(temporary):
raise FileExistsError(path)
if terminal_fault == "write":
raise OSError("fault_injected_terminal_receipt_write")
descriptor = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(descriptor, "wb") as handle:
if handle.write(payload) != len(payload):
raise OSError("terminal_receipt_short_write")
handle.flush()
if terminal_fault == "fsync":
raise OSError("fault_injected_terminal_receipt_fsync")
os.fsync(handle.fileno())
os.chmod(temporary, 0o600)
temporary_readback = temporary.read_bytes()
if temporary_readback != payload or hashlib.sha256(temporary_readback).hexdigest() != expected_digest:
raise OSError("terminal_receipt_temporary_readback_failed")
if terminal_fault == "link":
raise OSError("fault_injected_terminal_receipt_link")
os.link(temporary, path)
created_final = True
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
directory_fd = os.open(path.parent, directory_flags)
os.fsync(directory_fd)
if terminal_fault == "readback":
raise OSError("fault_injected_terminal_receipt_readback")
readback = path.read_bytes()
if readback != payload or hashlib.sha256(readback).hexdigest() != expected_digest:
raise OSError("terminal_receipt_final_readback_failed")
if json.loads(readback.decode("utf-8")) != document:
raise OSError("terminal_receipt_document_readback_failed")
temporary.unlink()
os.fsync(directory_fd)
except BaseException:
try:
if created_final:
path.unlink(missing_ok=True)
temporary.unlink(missing_ok=True)
if directory_fd is None and path.parent.is_dir():
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0)
directory_fd = os.open(path.parent, directory_flags)
if directory_fd is not None:
os.fsync(directory_fd)
finally:
raise
finally:
if directory_fd is not None:
os.close(directory_fd)
PY
then
return 1
fi
[ -f "$TERMINAL_RECEIPT_PATH" ] && [ ! -L "$TERMINAL_RECEIPT_PATH" ] || return 1
return 0
}
record_prestate() {
local name dest_path digest metadata
: > "$ROLLBACK_DIR/prestate.tsv"
chmod 600 "$ROLLBACK_DIR/prestate.tsv"
: > "$ROLLBACK_DIR/prestate.tsv" || return 74
chmod 600 "$ROLLBACK_DIR/prestate.tsv" || return 74
for name in "${PAYLOAD_FILES[@]}"; do
dest_path="$(destination_path "$name")"
if [ -f "$dest_path" ] && [ ! -L "$dest_path" ]; then
[ "$(stat -c '%U:%G' "$dest_path")" = "wooo:wooo" ] || return 71
FILE_EXISTED[$name]=1
digest="$(working_digest "$dest_path")"
metadata="$(stat -c '%u:%g:%a' "$dest_path")"
digest="$(working_digest "$dest_path")" || return 72
metadata="$(stat -c '%u:%g:%a' "$dest_path")" || return 72
PREVIOUS_DIGEST[$name]="$digest"
PREVIOUS_METADATA[$name]="$metadata"
printf '%s\t1\t%s\t%s\n' "$name" "$digest" "$metadata" >> "$ROLLBACK_DIR/prestate.tsv"
cp -p -- "$dest_path" "$ROLLBACK_DIR/$name"
printf '%s\t1\t%s\t%s\n' "$name" "$digest" "$metadata" >> "$ROLLBACK_DIR/prestate.tsv" || return 72
cp -p -- "$dest_path" "$ROLLBACK_DIR/$name" || return 72
[ "$(working_digest "$ROLLBACK_DIR/$name")" = "$digest" ] || return 72
elif [ ! -e "$dest_path" ]; then
FILE_EXISTED[$name]=0
PREVIOUS_DIGEST[$name]="-"
PREVIOUS_METADATA[$name]="-"
printf '%s\t0\t-\t-\n' "$name" >> "$ROLLBACK_DIR/prestate.tsv"
printf '%s\t0\t-\t-\n' "$name" >> "$ROLLBACK_DIR/prestate.tsv" || return 72
else
return 73
fi
done
fsync_rollback_prestate || return 75
}
register_run_owned_temp() {
RUN_OWNED_TEMP_PATHS["$1"]=1
}
verify_run_owned_temp_absence() {
local temporary
for temporary in "${!RUN_OWNED_TEMP_PATHS[@]}"; do
[ ! -e "$temporary" ] && [ ! -L "$temporary" ] || return 1
done
}
cleanup_run_owned_temps() {
local temporary failed=0 preserved=0
local fault=""
if [ "$FAULT_INJECTION_ENABLED" -eq 1 ]; then
fault="${HOST110_BACKUP_RUNTIME_TEST_ROLLBACK_FAULT:-}"
fi
for temporary in "${!RUN_OWNED_TEMP_PATHS[@]}"; do
if [ "$fault" = "preserve_first_temp" ] \
&& [ "$preserved" -eq 0 ] \
&& { [ -e "$temporary" ] || [ -L "$temporary" ]; }; then
preserved=1
failed=1
continue
fi
rm -f -- "$temporary" || failed=1
done
verify_run_owned_temp_absence || failed=1
[ "$failed" -eq 0 ]
}
rollback_transaction() {
local name dest_path destination_parent backup_path temporary failed=0
ROLLBACK_ATTEMPTED=1
ROLLBACK_PERFORMED=0
ROLLBACK_VERIFIED=0
RUN_TEMP_RESIDUE_VERIFIED=0
set +e
cleanup_run_owned_temps || failed=1
for name in "${PAYLOAD_FILES[@]}"; do
[ -n "${FILE_EXISTED[$name]+x}" ] || { failed=1; continue; }
dest_path="$(destination_path "$name")"
@@ -300,13 +664,22 @@ rollback_transaction() {
backup_path="$ROLLBACK_DIR/$name"
destination_parent="$(dirname "$dest_path")"
temporary="$destination_parent/.${name}.agent99-rollback-${RUN_ID}"
cp -p -- "$backup_path" "$temporary" \
&& mv -f -- "$temporary" "$dest_path" \
|| failed=1
register_run_owned_temp "$temporary"
if cp -p -- "$backup_path" "$temporary" && mv -f -- "$temporary" "$dest_path"; then
ROLLBACK_PERFORMED=1
else
failed=1
fi
else
rm -f -- "$dest_path" || failed=1
if rm -f -- "$dest_path"; then
ROLLBACK_PERFORMED=1
else
failed=1
fi
fi
done
cleanup_run_owned_temps || failed=1
fsync_destination_state || failed=1
for name in "${PAYLOAD_FILES[@]}"; do
dest_path="$(destination_path "$name")"
if [ "${FILE_EXISTED[$name]:-}" = "1" ]; then
@@ -320,8 +693,14 @@ rollback_transaction() {
failed=1
fi
done
if verify_run_owned_temp_absence; then
RUN_TEMP_RESIDUE_VERIFIED=1
else
failed=1
RUN_TEMP_RESIDUE_VERIFIED=0
fi
set -e
if [ "$failed" -eq 0 ]; then
if [ "$failed" -eq 0 ] && [ "$ROLLBACK_PERFORMED" -eq 1 ] && [ "$RUN_TEMP_RESIDUE_VERIFIED" -eq 1 ]; then
ROLLBACK_VERIFIED=1
return 0
fi
@@ -329,37 +708,78 @@ rollback_transaction() {
return 1
}
handle_agent99_signal() {
local signal_name="$1"
local signal_exit_code="$2"
if [ "$COMMIT_CRITICAL" -eq 1 ] || [ "$APPLY_COMPLETE" -eq 1 ]; then
if [ -z "$DEFERRED_SIGNAL" ]; then
DEFERRED_SIGNAL="$signal_name"
DEFERRED_SIGNAL_EXIT="$signal_exit_code"
fi
return 0
fi
exit "$signal_exit_code"
}
cleanup_postcommit_directory() {
local cleanup_kind="$1"
local cleanup_path="$2"
local fault=""
if [ "$FAULT_INJECTION_ENABLED" -eq 1 ]; then
fault="${HOST110_BACKUP_RUNTIME_TEST_CLEANUP_FAULT:-}"
fi
if [ "$fault" = "$cleanup_kind" ]; then
return 1
fi
rm -rf -- "$cleanup_path" || return 1
[ ! -e "$cleanup_path" ] && [ ! -L "$cleanup_path" ] || return 1
fsync_paths_and_parents "$cleanup_path" || return 1
[ ! -e "$cleanup_path" ] && [ ! -L "$cleanup_path" ]
}
cleanup() {
local status=$?
local exit_status=$?
local rollback_status="rollback_unverified"
local failure_receipt_written=0
trap - EXIT HUP INT TERM
if [ "$APPLY_STARTED" -eq 1 ] && [ "$APPLY_COMPLETE" -ne 1 ]; then
if rollback_transaction; then
rollback_status="failed_rolled_back"
fi
write_receipt "$rollback_status" "$ROLLBACK_VERIFIED" >/dev/null || true
if write_receipt "$rollback_status" "$ROLLBACK_ATTEMPTED" "$ROLLBACK_PERFORMED" \
"$ROLLBACK_VERIFIED" "$RUN_TEMP_RESIDUE_VERIFIED"; then
failure_receipt_written=1
fi
fi
[ -z "$STAGE_DIR" ] || rm -rf -- "$STAGE_DIR"
if [ "$ROLLBACK_VERIFIED" -eq 1 ]; then
[ -z "$ROLLBACK_DIR" ] || rm -rf -- "$ROLLBACK_DIR"
if [ "$APPLY_COMPLETE" -ne 1 ] && [ -n "$STAGE_DIR" ]; then
rm -rf -- "$STAGE_DIR" || true
fi
exit "$status"
if [ "$APPLY_COMPLETE" -ne 1 ] \
&& [ "$ROLLBACK_VERIFIED" -eq 1 ] \
&& [ "$failure_receipt_written" -eq 1 ]; then
if [ -n "$ROLLBACK_DIR" ]; then
rm -rf -- "$ROLLBACK_DIR" || true
fi
fi
exit "$exit_status"
}
validate_source || fail "source_validation_failed"
if [ "$MODE" = "check" ]; then
printf '{"schemaVersion":"agent99_host110_backup_runtime_executor_v1","mode":"check","ok":true,"sourceRevision":"%s","sourceHead":"%s","fileCount":18,"backupScriptFileCount":17,"backupHealthExporterIncluded":true,"remoteWritePerformed":false}\n' "$SOURCE_REVISION" "$SOURCE_HEAD"
printf '{"schemaVersion":"agent99_host110_backup_runtime_executor_v1","mode":"check","ok":true,"sourceRevision":"%s","sourceHead":"%s","runId":"%s","fileCount":18,"backupScriptFileCount":17,"backupHealthExporterIncluded":true,"remoteWritePerformed":false}\n' "$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID"
exit 0
fi
if [ "$MODE" = "verify" ]; then
verify_destination || fail "destination_verification_failed"
printf '{"schemaVersion":"agent99_host110_backup_runtime_executor_v1","mode":"verify","ok":true,"sourceRevision":"%s","sourceHead":"%s","fileCount":18,"backupScriptFileCount":17,"backupHealthExporterIncluded":true,"remoteWritePerformed":false}\n' "$SOURCE_REVISION" "$SOURCE_HEAD"
printf '{"schemaVersion":"agent99_host110_backup_runtime_executor_v1","mode":"verify","ok":true,"sourceRevision":"%s","sourceHead":"%s","runId":"%s","fileCount":18,"backupScriptFileCount":17,"backupHealthExporterIncluded":true,"remoteWritePerformed":false}\n' "$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID"
exit 0
fi
[ ! -e "$STATUS_ROOT/agent99-backup-runtime-deploy-${RUN_ID}.json" ] || fail "run_identity_receipt_exists"
[ ! -e "$RECEIPT_PATH" ] && [ ! -L "$RECEIPT_PATH" ] || fail "run_identity_receipt_exists"
[ ! -e "$TERMINAL_RECEIPT_PATH" ] && [ ! -L "$TERMINAL_RECEIPT_PATH" ] \
|| fail "run_identity_terminal_receipt_exists"
STAGE_DIR="$STAGE_ROOT/$RUN_ID"
ROLLBACK_DIR="$ROLLBACK_ROOT/$RUN_ID"
[ ! -e "$STAGE_DIR" ] || fail "run_identity_stage_exists"
@@ -380,8 +800,9 @@ fi
install -d -m 700 "$STAGE_DIR" "$ROLLBACK_DIR"
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM HUP
trap 'handle_agent99_signal INT 130' INT
trap 'handle_agent99_signal HUP 129' HUP
trap 'handle_agent99_signal TERM 143' TERM
for name in "${PAYLOAD_FILES[@]}"; do
install -m 700 "$SOURCE_STAGE/$name" "$STAGE_DIR/$name"
@@ -399,10 +820,17 @@ for name in "${PAYLOAD_FILES[@]}"; do
dest_path="$(destination_path "$name")"
destination_parent="$(dirname "$dest_path")"
temporary="$destination_parent/.${name}.agent99-${RUN_ID}"
register_run_owned_temp "$temporary"
install -m 755 "$STAGE_DIR/$name" "$temporary"
if [ "$FAULT_INJECTION_ENABLED" -eq 1 ] \
&& [ "${HOST110_BACKUP_RUNTIME_TEST_APPLY_FAULT:-}" = "after_temp_install" ] \
&& [ "$name" = "${PAYLOAD_FILES[0]}" ]; then
fail "fault_injected_after_temp_install"
fi
mv -f -- "$temporary" "$dest_path"
done
fsync_destination_state || fail "post_apply_durability_failed"
verify_destination || fail "post_apply_verification_failed"
verifier_result="$(BACKUP_RUNTIME_DEPLOY_CONTEXT=1 python3 "$SOURCE_STAGE/verify-host110-backup-runtime.py" \
--manifest "$SOURCE_STAGE/manifest.json" \
@@ -433,27 +861,118 @@ if not (
raise SystemExit(1)
PY
verify_run_owned_temp_absence || fail "post_apply_temp_residue_detected"
RUN_TEMP_RESIDUE_VERIFIED=1
COMMIT_CRITICAL=1
if ! write_receipt "payload_verified" 0 0 0 "$RUN_TEMP_RESIDUE_VERIFIED"; then
COMMIT_CRITICAL=0
fail "durable_receipt_failed"
fi
if [ "$FAULT_INJECTION_ENABLED" -eq 1 ] \
&& [ -n "${HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL:-}" ]; then
case "$HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL" in
INT|HUP|TERM) kill -s "$HOST110_BACKUP_RUNTIME_TEST_COMMIT_SIGNAL" "$$" ;;
esac
fi
APPLY_COMPLETE=1
receipt_path="$(write_receipt "verified" 0)"
rm -rf -- "$STAGE_DIR" "$ROLLBACK_DIR"
STAGE_DIR=""
ROLLBACK_DIR=""
trap - EXIT HUP INT TERM
python3 - "$SOURCE_REVISION" "$SOURCE_HEAD" "$receipt_path" "$verifier_result" <<'PY'
COMMIT_CRITICAL=0
stage_cleanup_verified=0
rollback_prestate_cleanup_verified=0
if cleanup_postcommit_directory "stage" "$STAGE_DIR"; then
stage_cleanup_verified=1
fi
if cleanup_postcommit_directory "rollback_prestate" "$ROLLBACK_DIR"; then
rollback_prestate_cleanup_verified=1
fi
terminal_status="verified"
terminal_exit_code=0
# Cleanup has completed. Freeze terminal signal state before the immutable
# terminal writer so its receipt, stdout, and exit status cannot diverge.
trap '' HUP INT TERM
if [ "$stage_cleanup_verified" -ne 1 ] || [ "$rollback_prestate_cleanup_verified" -ne 1 ]; then
terminal_status="cleanup_pending"
terminal_exit_code=75
elif [ -n "$DEFERRED_SIGNAL" ]; then
terminal_status="committed_signal_deferred"
terminal_exit_code="$DEFERRED_SIGNAL_EXIT"
fi
if ! write_terminal_receipt "$terminal_status" "$stage_cleanup_verified" \
"$rollback_prestate_cleanup_verified" "$DEFERRED_SIGNAL" "$terminal_exit_code"; then
trap - EXIT
python3 - "$SOURCE_REVISION" "$SOURCE_HEAD" "$RUN_ID" "$RECEIPT_PATH" \
"$TERMINAL_RECEIPT_PATH" "$stage_cleanup_verified" \
"$rollback_prestate_cleanup_verified" "$DEFERRED_SIGNAL" <<'PY'
import json
import sys
print(json.dumps({
"schemaVersion": "agent99_host110_backup_runtime_executor_v1",
"mode": "apply",
"ok": True,
"status": "terminal_receipt_failed",
"ok": False,
"sourceRevision": sys.argv[1],
"sourceHead": sys.argv[2],
"runId": sys.argv[3],
"fileCount": 18,
"backupScriptFileCount": 17,
"backupHealthExporterIncluded": True,
"payloadCommitted": True,
"rollbackAttempted": False,
"rollbackPerformed": False,
"receipt": sys.argv[3],
"verifier": json.loads(sys.argv[4]),
"rollbackVerified": False,
"zeroResidueVerified": True,
"zeroResidueScope": "run_owned_destination_temps",
"stageCleanupVerified": sys.argv[6] == "1",
"rollbackPrestateCleanupVerified": sys.argv[7] == "1",
"receipt": sys.argv[4],
"terminalReceipt": sys.argv[5],
"terminalReceiptPublished": False,
"terminalExitCode": 76,
"deferredSignal": sys.argv[8] or None,
"independentVerifierVerified": True,
"verifier": None,
}, ensure_ascii=True, sort_keys=True))
PY
printf 'HOST110_BACKUP_RUNTIME_OK=0\nERROR=terminal_receipt_failed\n' >&2
exit 76
fi
if [ "$stage_cleanup_verified" -eq 1 ]; then STAGE_DIR=""; fi
if [ "$rollback_prestate_cleanup_verified" -eq 1 ]; then ROLLBACK_DIR=""; fi
trap - EXIT
python3 - "$SOURCE_REVISION" "$SOURCE_HEAD" "$RECEIPT_PATH" "$TERMINAL_RECEIPT_PATH" \
"$verifier_result" "$stage_cleanup_verified" "$rollback_prestate_cleanup_verified" \
"$terminal_status" "$terminal_exit_code" "$DEFERRED_SIGNAL" "$RUN_ID" <<'PY'
import json
import sys
status = sys.argv[8]
verifier = json.loads(sys.argv[5])
print(json.dumps({
"schemaVersion": "agent99_host110_backup_runtime_executor_v1",
"mode": "apply",
"status": status,
"ok": status == "verified",
"sourceRevision": sys.argv[1],
"sourceHead": sys.argv[2],
"runId": sys.argv[11],
"fileCount": 18,
"backupScriptFileCount": 17,
"backupHealthExporterIncluded": True,
"payloadCommitted": True,
"rollbackAttempted": False,
"rollbackPerformed": False,
"rollbackVerified": False,
"zeroResidueVerified": True,
"zeroResidueScope": "run_owned_destination_temps",
"stageCleanupVerified": sys.argv[6] == "1",
"rollbackPrestateCleanupVerified": sys.argv[7] == "1",
"receipt": sys.argv[3],
"terminalReceipt": sys.argv[4],
"terminalReceiptPublished": True,
"terminalExitCode": int(sys.argv[9]),
"deferredSignal": sys.argv[10] or None,
"independentVerifierVerified": True,
"verifier": verifier if status == "verified" else None,
}, ensure_ascii=True, sort_keys=True))
PY
exit "$terminal_exit_code"

View File

@@ -27,11 +27,46 @@ RUNTIME_LOCK = Path("/tmp/agent99-host110-backup-runtime.lock")
def acquire_runtime_lock():
if Path(__file__).resolve().parent != Path("/backup/scripts") or os.environ.get("BACKUP_RUNTIME_SHARED_LOCK_HELD") == "1":
if Path(__file__).resolve().parent != Path("/backup/scripts"):
return None
if RUNTIME_LOCK.is_symlink():
raise RuntimeError("backup_runtime_gate_unsafe")
handle = RUNTIME_LOCK.open("a+")
inherited_fd_path: Path | None = None
proc_fd_root = Path(f"/proc/{os.getpid()}/fd")
if proc_fd_root.is_dir():
candidate = proc_fd_root / "197"
if candidate.exists() or candidate.is_symlink():
inherited_fd_path = candidate
else:
candidate = Path("/dev/fd/197")
if candidate.exists() or candidate.is_symlink():
inherited_fd_path = candidate
if inherited_fd_path is None:
handle = RUNTIME_LOCK.open("a+")
else:
if not RUNTIME_LOCK.is_file():
raise RuntimeError("backup_runtime_gate_unavailable")
if proc_fd_root.is_dir():
try:
inherited_target = inherited_fd_path.resolve(strict=True)
expected_target = RUNTIME_LOCK.resolve(strict=True)
except OSError as exc:
raise RuntimeError("backup_runtime_inherited_fd_unresolved") from exc
else:
try:
raw_target = fcntl.fcntl(197, getattr(fcntl, "F_GETPATH", 50), b"\0" * 1024)
inherited_target = Path(os.fsdecode(raw_target.split(b"\0", 1)[0])).resolve(strict=True)
expected_target = RUNTIME_LOCK.resolve(strict=True)
except OSError as exc:
raise RuntimeError("backup_runtime_inherited_fd_unresolved") from exc
if inherited_target != expected_target:
raise RuntimeError("backup_runtime_inherited_fd_mismatch")
try:
handle = os.fdopen(os.dup(197), "a+")
except OSError as exc:
raise RuntimeError("backup_runtime_inherited_fd_unavailable") from exc
if RUNTIME_LOCK.stat().st_uid != os.getuid():
handle.close()
raise RuntimeError("backup_runtime_gate_owner_invalid")

View File

@@ -0,0 +1,409 @@
$ErrorActionPreference = "Stop"
function Write-ReplayTrace {
param([string]$Step)
if ($ReplayTrace) { Write-Output "REPLAY_STEP=$Step" }
}
if (-not $BrokerGzipBase64) { throw "broker_payload_missing" }
Write-ReplayTrace "decode_source"
$compressed = [Convert]::FromBase64String([string]$BrokerGzipBase64)
$inputStream = New-Object IO.MemoryStream(,$compressed)
$gzip = New-Object IO.Compression.GzipStream(
$inputStream,
[IO.Compression.CompressionMode]::Decompress
)
$reader = New-Object IO.StreamReader($gzip, [Text.Encoding]::UTF8)
try {
$brokerSource = $reader.ReadToEnd()
} finally {
$reader.Dispose()
$gzip.Dispose()
$inputStream.Dispose()
}
$tokens = $null
$parseErrors = $null
$ast = [Management.Automation.Language.Parser]::ParseInput(
$brokerSource,
[ref]$tokens,
[ref]$parseErrors
)
if ($parseErrors.Count -ne 0) { throw "broker_parse_failed" }
Write-ReplayTrace "define_functions"
$functionNames = @(
"Test-Agent99JsonField",
"Assert-Agent99JsonFields",
"Assert-Agent99ExactJsonFields",
"Convert-Agent99ExecutorDocument",
"Assert-Agent99ApplyResultFields",
"Assert-Agent99VerifierFields",
"Get-Agent99ExecutorResult",
"Get-Agent99CommittedFailureResult",
"Assert-Agent99VerifierEvidence",
"Assert-Agent99PreflightEvidence",
"Get-Agent99TransportReceiptReadback",
"New-Agent99TransportLossReconciliation",
"Assert-Agent99ExecutorEvidence",
"Assert-Agent99ReconciliationEvidence",
"Assert-Agent99ExistingBrokerEvidence"
)
foreach ($functionName in $functionNames) {
$node = $ast.Find(
{
param($candidate)
$candidate -is [Management.Automation.Language.FunctionDefinitionAst] -and
$candidate.Name -eq $functionName
},
$true
)
if ($null -eq $node) { throw "broker_function_missing_$functionName" }
Invoke-Expression $node.Extent.Text
}
$SourceRevision = "a" * 40
$RunId = "windows99-contract-replay"
$Mode = "Apply"
$TargetHost = "192.168.0.110"
$ExpectedFileCount = 18
$sourceHead = "b" * 40
$VerifierSha256 = "c" * 64
$payloadReceipt = "/backup/status/agent99-backup-runtime-deploy-$RunId.json"
$terminalReceipt = "/backup/status/agent99-backup-runtime-terminal-$RunId.json"
function Copy-ReplayDocument {
param([object]$Document)
return ($Document | ConvertTo-Json -Compress -Depth 12 | ConvertFrom-Json)
}
function New-ReplayTransport {
param([object]$Document, [int]$ExitCode)
return [pscustomobject]@{
ok = [bool]($ExitCode -eq 0)
exitCode = $ExitCode
reason = if ($ExitCode -eq 0) { "completed" } else { "process_failed" }
stdout = $Document | ConvertTo-Json -Compress -Depth 12
stderrPresent = $false
}
}
function Assert-Rejected {
param([scriptblock]$Operation, [string]$Name)
$accepted = $false
try {
& $Operation | Out-Null
$accepted = $true
} catch {}
if ($accepted) { throw "contract_mutation_accepted_$Name" }
}
$verifier = [pscustomobject]@{
schemaVersion = "agent99_host110_backup_runtime_verifier_v1"
ok = $true
sourceRevision = $SourceRevision
runId = $RunId
fileCount = 18
backupScriptFileCount = 17
backupHealthExporterIncluded = $true
destination = "/backup/scripts"
exporterDestination = "/home/wooo/scripts/backup-health-textfile-exporter.py"
remoteWritePerformed = $false
secretValuesRead = $false
selfIdentityVerified = $true
verifierSha256 = $VerifierSha256
}
$success = [pscustomobject]@{
schemaVersion = "agent99_host110_backup_runtime_executor_v1"
mode = "apply"
status = "verified"
ok = $true
sourceRevision = $SourceRevision
sourceHead = $sourceHead
runId = $RunId
fileCount = 18
backupScriptFileCount = 17
backupHealthExporterIncluded = $true
payloadCommitted = $true
rollbackAttempted = $false
rollbackPerformed = $false
rollbackVerified = $false
zeroResidueVerified = $true
zeroResidueScope = "run_owned_destination_temps"
stageCleanupVerified = $true
rollbackPrestateCleanupVerified = $true
receipt = $payloadReceipt
terminalReceipt = $terminalReceipt
terminalReceiptPublished = $true
terminalExitCode = 0
deferredSignal = $null
independentVerifierVerified = $true
verifier = $verifier
}
Write-ReplayTrace "accept_success"
Get-Agent99ExecutorResult (New-ReplayTransport $success 0) "Apply" $VerifierSha256 | Out-Null
$topLevelRequired = @(
"schemaVersion", "mode", "status", "ok", "sourceRevision", "sourceHead",
"runId", "fileCount", "backupScriptFileCount", "backupHealthExporterIncluded",
"payloadCommitted", "rollbackAttempted", "rollbackPerformed", "rollbackVerified",
"zeroResidueVerified", "zeroResidueScope", "stageCleanupVerified",
"rollbackPrestateCleanupVerified", "receipt", "terminalReceipt", "terminalExitCode",
"terminalReceiptPublished", "deferredSignal", "independentVerifierVerified", "verifier"
)
$verifierRequired = @(
"schemaVersion", "ok", "sourceRevision", "runId", "fileCount",
"backupScriptFileCount", "backupHealthExporterIncluded", "remoteWritePerformed",
"secretValuesRead", "selfIdentityVerified", "destination", "exporterDestination",
"verifierSha256"
)
$missingFieldRejections = 0
Write-ReplayTrace "reject_missing_top"
foreach ($field in $topLevelRequired) {
$mutated = Copy-ReplayDocument $success
$mutated.PSObject.Properties.Remove($field)
Assert-Rejected { Get-Agent99ExecutorResult (New-ReplayTransport $mutated 0) "Apply" $VerifierSha256 } "missing_$field"
$missingFieldRejections++
}
Write-ReplayTrace "reject_missing_verifier"
foreach ($field in $verifierRequired) {
$mutated = Copy-ReplayDocument $success
$mutated.verifier.PSObject.Properties.Remove($field)
Assert-Rejected { Get-Agent99ExecutorResult (New-ReplayTransport $mutated 0) "Apply" $VerifierSha256 } "missing_verifier_$field"
$missingFieldRejections++
}
$typeMutations = @(
@{ field = "rollbackAttempted"; value = "false" },
@{ field = "rollbackPerformed"; value = "false" },
@{ field = "rollbackVerified"; value = "false" },
@{ field = "terminalExitCode"; value = "0" },
@{ field = "deferredSignal"; value = 0 }
)
$typeRejections = 0
Write-ReplayTrace "reject_types"
foreach ($mutation in $typeMutations) {
$mutated = Copy-ReplayDocument $success
$mutated.($mutation.field) = $mutation.value
Assert-Rejected { Get-Agent99ExecutorResult (New-ReplayTransport $mutated 0) "Apply" $VerifierSha256 } "type_$($mutation.field)"
$typeRejections++
}
$mutated = Copy-ReplayDocument $success
$mutated.verifier.remoteWritePerformed = "false"
Assert-Rejected { Get-Agent99ExecutorResult (New-ReplayTransport $mutated 0) "Apply" $VerifierSha256 } "type_verifier_remoteWritePerformed"
$typeRejections++
$cleanupPending = Copy-ReplayDocument $success
$cleanupPending.ok = $false
$cleanupPending.status = "cleanup_pending"
$cleanupPending.stageCleanupVerified = $false
$cleanupPending.terminalExitCode = 75
$cleanupPending.verifier = $null
Write-ReplayTrace "accept_cleanup_pending"
Get-Agent99CommittedFailureResult (New-ReplayTransport $cleanupPending 75) | Out-Null
$signalDeferred = Copy-ReplayDocument $success
$signalDeferred.ok = $false
$signalDeferred.status = "committed_signal_deferred"
$signalDeferred.terminalExitCode = 143
$signalDeferred.deferredSignal = "TERM"
$signalDeferred.verifier = $null
Write-ReplayTrace "accept_signal_deferred"
Get-Agent99CommittedFailureResult (New-ReplayTransport $signalDeferred 143) | Out-Null
$terminalReceiptFailed = Copy-ReplayDocument $success
$terminalReceiptFailed.ok = $false
$terminalReceiptFailed.status = "terminal_receipt_failed"
$terminalReceiptFailed.terminalReceiptPublished = $false
$terminalReceiptFailed.terminalExitCode = 76
$terminalReceiptFailed.verifier = $null
Write-ReplayTrace "accept_terminal_receipt_failed"
Get-Agent99CommittedFailureResult (New-ReplayTransport $terminalReceiptFailed 76) | Out-Null
$EvidencePath = "C:\Wooo\Agent99\evidence\host110-backup-runtime\agent99-host110-backup-runtime-$RunId.json"
$preflight = [pscustomobject]@{
schemaVersion = "agent99_host110_backup_runtime_preflight_v1"
ok = $true
sourceRevision = $SourceRevision
sourceHead = $sourceHead
fileCount = 18
backupScriptFileCount = 17
backupHealthExporterIncluded = $true
remoteWritePerformed = $false
}
$brokerSuccess = [pscustomobject]@{
schemaVersion = "agent99_host110_backup_runtime_broker_receipt_v3"
ok = $true
status = "verified"
mode = "Apply"
sourceRevision = $SourceRevision
sourceHead = $sourceHead
runId = $RunId
targetHost = $TargetHost
sourceTransport = "windows99_gitea_exact_revision_manifest"
executor = "host110_backup_runtime_executor"
verifier = "independent_host110_backup_runtime_python_readback"
decisionProvider = "deterministic_only"
criticProvider = "deterministic_only"
agentAction = "controlled_apply"
check = $preflight
apply = $success
verify = $verifier
cleanupVerified = $true
errorCode = ""
elapsedSeconds = 1.25
evidence = $EvidencePath
secretValuesRead = $false
rawSessionStored = $false
}
Write-ReplayTrace "accept_existing_success"
Assert-Agent99ExistingBrokerEvidence $brokerSuccess $EvidencePath
$brokerRequired = @(
"schemaVersion", "ok", "status", "mode", "sourceRevision", "sourceHead",
"runId", "targetHost", "sourceTransport", "executor", "verifier",
"decisionProvider", "criticProvider", "agentAction", "check", "apply",
"verify", "cleanupVerified", "errorCode", "elapsedSeconds", "evidence",
"secretValuesRead", "rawSessionStored"
)
$existingEvidenceRejections = 0
Write-ReplayTrace "reject_existing_missing"
foreach ($field in $brokerRequired) {
$mutated = Copy-ReplayDocument $brokerSuccess
$mutated.PSObject.Properties.Remove($field)
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_missing_$field"
$existingEvidenceRejections++
}
$existingMutations = @(
@{ name = "ok_string_false"; field = "ok"; value = "false" },
@{ name = "wrong_mode"; field = "mode"; value = "Verify" },
@{ name = "wrong_run_id"; field = "runId"; value = "other-run" },
@{ name = "wrong_status"; field = "status"; value = "readback_ok" },
@{ name = "wrong_cleanup"; field = "cleanupVerified"; value = $false },
@{ name = "cleanup_string"; field = "cleanupVerified"; value = "true" },
@{ name = "elapsed_string"; field = "elapsedSeconds"; value = "1.25" }
)
Write-ReplayTrace "reject_existing_conflicts"
foreach ($mutation in $existingMutations) {
$mutated = Copy-ReplayDocument $brokerSuccess
$mutated.($mutation.field) = $mutation.value
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_$($mutation.name)"
$existingEvidenceRejections++
}
$mutated = Copy-ReplayDocument $brokerSuccess
$mutated.apply.runId = "other-run"
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_apply_run_conflict"
$existingEvidenceRejections++
$mutated = Copy-ReplayDocument $brokerSuccess
$mutated.verify.PSObject.Properties.Remove("selfIdentityVerified")
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_verify_nested_missing"
$existingEvidenceRejections++
$mutated = Copy-ReplayDocument $brokerSuccess
$mutated | Add-Member -NotePropertyName unexpected -NotePropertyValue $true
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_extra_top"
$existingEvidenceRejections++
$mutated = Copy-ReplayDocument $brokerSuccess
$mutated.apply | Add-Member -NotePropertyName unexpected -NotePropertyValue $true
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $mutated $EvidencePath } "existing_extra_apply"
$existingEvidenceRejections++
$brokerFailure = Copy-ReplayDocument $brokerSuccess
$brokerFailure.ok = $false
$brokerFailure.status = "failed"
$brokerFailure.cleanupVerified = $false
$brokerFailure.errorCode = "host110_apply_cleanup_pending"
$brokerFailure.apply = $cleanupPending
$brokerFailure.verify = $null
Write-ReplayTrace "accept_existing_failure"
Assert-Agent99ExistingBrokerEvidence $brokerFailure $EvidencePath
$failureWithoutPreflight = Copy-ReplayDocument $brokerFailure
$failureWithoutPreflight.check = $null
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $failureWithoutPreflight $EvidencePath } "failure_preflight_missing"
$existingEvidenceRejections++
$failureWithImpossibleVerifier = Copy-ReplayDocument $brokerFailure
$failureWithImpossibleVerifier.verify = $verifier
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $failureWithImpossibleVerifier $EvidencePath } "failure_verifier_conflict"
$existingEvidenceRejections++
$postcommitCleanupFailure = Copy-ReplayDocument $brokerSuccess
$postcommitCleanupFailure.ok = $false
$postcommitCleanupFailure.status = "failed"
$postcommitCleanupFailure.cleanupVerified = $false
$postcommitCleanupFailure.errorCode = "remote_source_stage_cleanup_failed"
Assert-Agent99ExistingBrokerEvidence $postcommitCleanupFailure $EvidencePath
$malformedFailure = Copy-ReplayDocument $brokerFailure
$malformedFailure.ok = "false"
Assert-Rejected { Assert-Agent99ExistingBrokerEvidence $malformedFailure $EvidencePath } "malformed_failure_not_upgraded"
$existingEvidenceRejections++
$sourcePackage = [pscustomobject]@{ sourceHead = $sourceHead }
$transportLoss = [pscustomobject]@{ reason = "transport_timeout"; exitCode = -2 }
$receiptReadback = [pscustomobject]@{
schemaVersion = "agent99_host110_backup_transport_receipt_readback_v1"
ok = $true
sourceRevision = $SourceRevision
sourceHead = $sourceHead
runId = $RunId
payloadReceipt = $payloadReceipt
payloadState = "valid"
terminalReceipt = $terminalReceipt
terminalState = "missing"
terminalStatus = ""
terminalCleanupVerified = $false
terminalExitCode = -1
remoteWritePerformed = $false
secretValuesRead = $false
}
Write-ReplayTrace "reconcile_terminal_missing"
$terminalMissing = New-Agent99TransportLossReconciliation $transportLoss $sourcePackage $receiptReadback $verifier
if (
$terminalMissing.status -ne "committed_verified_terminal_missing" -or
-not $terminalMissing.payloadCommitted -or
-not $terminalMissing.destinationVerified -or
$terminalMissing.retryApplyAllowed
) { throw "transport_terminal_missing_reconciliation_failed" }
Assert-Agent99ReconciliationEvidence $terminalMissing $sourceHead
$terminalPresentReadback = Copy-ReplayDocument $receiptReadback
$terminalPresentReadback.terminalState = "valid"
$terminalPresentReadback.terminalStatus = "verified"
$terminalPresentReadback.terminalCleanupVerified = $true
$terminalPresentReadback.terminalExitCode = 0
$transportRecovered = New-Agent99TransportLossReconciliation $transportLoss $sourcePackage $terminalPresentReadback $verifier
if ($transportRecovered.status -ne "committed_verified_transport_lost" -or $transportRecovered.retryApplyAllowed) {
throw "transport_verified_reconciliation_failed"
}
Assert-Agent99ReconciliationEvidence $transportRecovered $sourceHead
$conflictingTransport = Copy-ReplayDocument $transportRecovered
$conflictingTransport.terminalStatus = "cleanup_pending"
Assert-Rejected { Assert-Agent99ReconciliationEvidence $conflictingTransport $sourceHead } "transport_terminal_status_conflict"
$existingEvidenceRejections++
$committedUnknown = New-Agent99TransportLossReconciliation `
$transportLoss $sourcePackage $null $null "receipt_readback_unavailable" "verifier_unavailable"
if (
$committedUnknown.status -ne "committed_unknown" -or
$committedUnknown.payloadCommitted -or
$committedUnknown.destinationVerified -or
$committedUnknown.retryApplyAllowed
) { throw "transport_unknown_reconciliation_failed" }
Assert-Agent99ReconciliationEvidence $committedUnknown $sourceHead
Write-ReplayTrace "complete"
[pscustomobject]@{
schemaVersion = "agent99_host110_broker_contract_replay_v1"
ok = $true
parser = "WindowsPowerShell"
successAccepted = $true
cleanupPendingPreserved = $true
committedSignalPreserved = $true
terminalReceiptFailurePreserved = $true
existingSuccessAccepted = $true
existingFailurePreserved = $true
existingEvidenceRejections = $existingEvidenceRejections
terminalMissingReconciled = $true
committedUnknownPreserved = $true
missingFieldRejections = $missingFieldRejections
typeRejections = $typeRejections
remoteWritePerformed = $false
secretValuesRead = $false
} | ConvertTo-Json -Compress