fix(agent99): fail closed host110 runtime replay
This commit is contained in:
Your Name
202
docs/proposals/AWOOOI-AG99-H110-BACKUP-RUNTIME-V11.md
Normal file
202
docs/proposals/AWOOOI-AG99-H110-BACKUP-RUNTIME-V11.md
Normal file
@@ -0,0 +1,202 @@
|
||||
# AWOOOI Agent99 Host110 Backup Runtime V11
|
||||
|
||||
Status: proposed. Central review and explicit owner approval are required before
|
||||
any production apply. This candidate must not be pushed to `main`, sent through
|
||||
CD, or applied to Windows99 or Host110 while it remains proposed.
|
||||
|
||||
This artifact supersedes V10. Every V10 identity below is void for approval and
|
||||
remains audit evidence only:
|
||||
|
||||
- branch/ref: `codex/host110-backup-runtime-v10-20260718`
|
||||
- candidate/revision: `d1797a1c8766722ae00ac14e20fed00f4e3f3a5b`
|
||||
- artifact SHA-256: `eb48661bbecc4e169111879bb18dfab63be29878d5f5c3651d9917f81f195d28`
|
||||
- exact diff SHA-256: `f7efb98d5f91fa6899713b692c8bb015fcb6ae6dba4291cc6ae429290072964e`
|
||||
- every other V10 proposal, content, revision, review, and derived hash
|
||||
|
||||
No V10 hash, ref, review, receipt, or test grants production execution
|
||||
authority.
|
||||
|
||||
## Exact Scope
|
||||
|
||||
- Agent99 runtime bundle: 19 files, guarded by one producer/consumer parity
|
||||
test.
|
||||
- Host110 backup payload: 18 ordered files, comprising 17 backup scripts and
|
||||
`backup-health-textfile-exporter.py`.
|
||||
- Host110 remote stage: 21 unique basenames, comprising the ordered 18-file
|
||||
payload, fixed executor, independent verifier, and `manifest.json`.
|
||||
- Fixed target: `wooo@192.168.0.110`, dispatched only by the Windows99 Agent99
|
||||
typed broker from an exact Gitea revision and digest-bound manifest.
|
||||
- V11 changes the Windows99 broker, three shell shared-lock entrypoints, the
|
||||
Host188 archive verifier's shared-lock acquisition, and their no-write replay
|
||||
tests.
|
||||
- `scripts/backup/backup-gitea.sh` has zero diff from the production base. No
|
||||
Gitea primary stop/start or backup-container creation is part of this apply.
|
||||
- The restore drill and offsite gate gain only canonical shared-lock behavior.
|
||||
Applying this candidate replaces those scripts but does not invoke a drill,
|
||||
run a backup, or start an offsite sync.
|
||||
- Prior proposal documents remain tracked audit artifacts and are not runtime
|
||||
authority.
|
||||
|
||||
## V10 Defects Closed
|
||||
|
||||
### Persisted Broker Evidence Is Exact And Fail-Closed
|
||||
|
||||
Before reusing a run-bound broker receipt, the broker now validates exact key
|
||||
sets, required native JSON types, schema version, run ID, source revision, mode,
|
||||
target, evidence path binding, status, error, cleanup, and nested Apply/Verify
|
||||
contracts. Missing, extra, wrong-type, wrong-mode, wrong-run, conflicting, or
|
||||
malformed evidence fails closed before any replay decision.
|
||||
|
||||
A non-empty string such as `"false"` can no longer be cast to a truthy Boolean.
|
||||
An existing failure remains a failure and cannot be upgraded to success merely
|
||||
because it is parseable or shares a source revision.
|
||||
|
||||
### Mandatory Post-Commit Transport Reconciliation
|
||||
|
||||
If Apply loses SSH transport or times out, the broker performs a read-only,
|
||||
exact-RunId receipt readback under `/backup/status` and an independent
|
||||
destination verifier readback. It never blindly retries Apply and never
|
||||
collapses a post-commit state into a generic precommit failure.
|
||||
|
||||
The reconciliation states are explicit and non-success:
|
||||
|
||||
- `committed_verified_terminal_missing`
|
||||
- `committed_verified_transport_lost`
|
||||
- `committed_cleanup_pending_transport_lost`
|
||||
- `committed_signal_deferred_transport_lost`
|
||||
- `committed_unknown`
|
||||
|
||||
Every state records `retryApplyAllowed=false`. Receipt schema, native types,
|
||||
source/run/head identity, payload paths and counts, verifier identity, cleanup,
|
||||
signal, and terminal semantics are validated before classification. The
|
||||
readback performs no remote write and reads no secret value.
|
||||
|
||||
### Canonical Shared Lock And Nested Re-entry
|
||||
|
||||
Production backup entrypoints canonicalize absolute, relative, and symlink
|
||||
invocation paths before deciding whether the Host110 runtime lock applies.
|
||||
Unresolvable production identity fails closed.
|
||||
|
||||
For nested calls, an inherited FD 197 is accepted only when its canonical
|
||||
target is the exact runtime lock. Linux uses `/proc/$$/fd/197`; the bounded
|
||||
non-Linux fallback resolves the descriptor path with `F_GETPATH`. The entrypoint
|
||||
then requests the shared lock on the same descriptor. It does not close and
|
||||
reopen a valid inherited descriptor, so there is no release/reacquire gap.
|
||||
|
||||
If FD 197 is absent, the entrypoint opens the lock and acquires the shared side.
|
||||
If FD 197 exists but points elsewhere or cannot be resolved, execution fails
|
||||
closed. `BACKUP_RUNTIME_SHARED_LOCK_HELD` and all other external environment
|
||||
values grant no lock authority.
|
||||
|
||||
The Host188 archive verifier follows the same contract: exact inherited FD 197
|
||||
is duplicated from the same open file description, wrong identity fails
|
||||
closed, and a forged environment value cannot bypass an active exclusive lock.
|
||||
|
||||
## Apply Effects
|
||||
|
||||
- Windows99 Agent99 runtime promotion writes the 19 runtime files plus bounded
|
||||
manifest and receipt state.
|
||||
- Host110 Apply replaces the ordered 18 payload paths through one exclusive,
|
||||
digest-bound filesystem transaction and writes payload/terminal receipts
|
||||
below `/backup/status`.
|
||||
- The deployed common library, restore drill, offsite gate, and Host188 archive
|
||||
verifier gain the shared-lock behavior described above. Apply does not invoke
|
||||
their backup, drill, archive, or sync behavior.
|
||||
- Apply does not stop/start Gitea, create a backup container, call Google Drive,
|
||||
prune snapshots, restart a VM or service, change a database/network/firewall,
|
||||
or read/change a credential or secret.
|
||||
|
||||
## Reproducible Validation
|
||||
|
||||
Expanded focused and compatibility suite:
|
||||
|
||||
```sh
|
||||
pytest -q \
|
||||
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
|
||||
scripts/backup/tests/test_verify_offsite_full_sync_runtime.py \
|
||||
scripts/backup/tests/test_verify_host188_products_archive.py \
|
||||
scripts/backup/tests/test_backup_host188_products_contract.py \
|
||||
scripts/ops/tests/test_agent99_signoz_metadata_executor_runtime_entrypoint.py \
|
||||
scripts/ops/tests/test_db_bounded_executor_runtime_entrypoint.py \
|
||||
scripts/reboot-recovery/tests/test_agent99_live_preflight_decision.py \
|
||||
scripts/reboot-recovery/tests/test_agent99_remote_atomic_deploy_transport_contract.py \
|
||||
scripts/reboot-recovery/tests/test_host112_manager_recovery_contract.py
|
||||
```
|
||||
|
||||
Current result: `120 passed, 4 skipped`.
|
||||
|
||||
The shared-lock subset includes real `fcntl` contention for absolute,
|
||||
relative, symlink, forged-environment, inherited-exact-FD, inherited-wrong-FD,
|
||||
and Host188 archive paths. Current result: `18 passed, 31 deselected`.
|
||||
|
||||
Static, syntax, and repository checks:
|
||||
|
||||
```sh
|
||||
ruff check \
|
||||
scripts/reboot-recovery/tests/test_agent99_host110_backup_runtime_broker.py \
|
||||
scripts/backup/verify-host188-products-archive.py
|
||||
bash -n \
|
||||
scripts/backup/common.sh \
|
||||
scripts/backup/gitea-full-backup-restore-drill.sh \
|
||||
scripts/backup/backup-offsite-readiness-gate.sh \
|
||||
scripts/backup/host110-backup-runtime-executor.sh
|
||||
git diff --check
|
||||
PATH=/Users/ogt/.pyenv/shims:$PATH bash scripts/ops/ansible-validate.sh
|
||||
```
|
||||
|
||||
`ansible-validate.sh` parses
|
||||
`ops/reboot-recovery/full-stack-backup-baseline.yml` as YAML metadata. It runs
|
||||
`ansible-playbook --syntax-check` only for actual playbooks. Current checks are
|
||||
green; `ansible-lint` is not installed and is reported as skipped.
|
||||
|
||||
Windows99 no-write PowerShell validation consists of:
|
||||
|
||||
1. parsing the complete candidate broker with Windows PowerShell;
|
||||
2. loading the candidate broker's actual function AST in the contract replay;
|
||||
3. replaying valid success, committed non-success, existing evidence rejection,
|
||||
terminal-missing reconciliation, and committed-unknown cases.
|
||||
|
||||
Current readback is parser `errorCount=0`, plus 38 existing-evidence
|
||||
rejections, 38 missing-field rejections, 6 type rejections,
|
||||
`terminalMissingReconciled=true`, `committedUnknownPreserved=true`,
|
||||
`remoteWritePerformed=false`, and `secretValuesRead=false`.
|
||||
|
||||
## Exact Identity And Diff Serialization
|
||||
|
||||
The central-review intake must bind all of the following independently:
|
||||
|
||||
- live Gitea feature ref `codex/host110-backup-runtime-v11-20260718`;
|
||||
- exact candidate Git SHA;
|
||||
- this tracked artifact path and exact artifact-bytes SHA-256;
|
||||
- exact binary diff SHA-256, path count, and insertion/deletion counts;
|
||||
- the 19/18/21 count contracts;
|
||||
- `scripts/backup/backup-gitea.sh` zero diff;
|
||||
- the Windows99 no-write replay result.
|
||||
|
||||
Exact diff bytes are serialized with:
|
||||
|
||||
```sh
|
||||
git diff --binary --full-index \
|
||||
e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3..<candidate_git_sha> |
|
||||
shasum -a 256
|
||||
```
|
||||
|
||||
No alternative diff serialization or untracked artifact bytes may be used for
|
||||
approval identity.
|
||||
|
||||
## Rollback
|
||||
|
||||
- Source rollback reverts the exact V11 diff to
|
||||
`e7e3bcf8a5a8a19c1253d73f51d9dca248af0aa3` without force-pushing.
|
||||
- Agent99 rollback restores prior runtime files and manifest and verifies prior
|
||||
hashes independently.
|
||||
- Before payload commit, Host110 rollback restores prior payload content,
|
||||
SHA-256, uid, gid, and mode under the exclusive lock and proves no run-owned
|
||||
destination temporary remains.
|
||||
- After payload commit, transport loss cannot trigger a blind Apply retry.
|
||||
Exact receipts and independent destination readback determine whether the
|
||||
state is committed or unknown; every uncertain state remains non-success.
|
||||
- Source rollback restores prior lock behavior. It does not execute a backup,
|
||||
restore drill, offsite sync, container lifecycle, or service restart.
|
||||
- Any identity, schema, type, receipt, reconciliation, lock, cleanup, or
|
||||
residue mismatch fails closed and cannot produce broker `verified`.
|
||||
Reference in New Issue
Block a user